Research Papers

Introduced backpropagation through time for training recurrent neural networks, enabling sequential data processing.

Proposed gating mechanisms (input, forget, output gates) to solve the vanishing gradient problem in RNNs, enabling learning over long sequences.

Developed efficient word embeddings via skip-gram and CBOW architectures, capturing semantic relationships in dense vector representations.

Introduced encoder-decoder RNN architectures for machine translation, establishing the pattern for generative sequence-to-sequence tasks.

Added attention to seq2seq models, allowing dynamic focus on relevant input parts rather than compressing everything into a fixed vector.

Introduced generative adversarial networks — two neural networks competing against each other (a generator creates fake data, a discriminator tries to detect fakes) which produces increasingly realistic outputs.

Introduced the Transformer architecture using self-attention mechanisms, replacing RNNs entirely. Enabled parallel training and superior long-range dependency modeling.

First decoder-only Transformer pretrained generatively on BooksCorpus. Demonstrated zero-shot transfer learning via fine-tuning.

Encoder-only bidirectional pretraining with masked language modeling (MLM) and next-sentence prediction. Set SOTA on GLUE benchmarks.

Framed all NLP tasks as text-to-text problems. Scaled pretraining and fine-tuning systematically across tasks.

Scaled GPT to 1.5B parameters on WebText. Demonstrated emergent unsupervised multitask learning without task-specific fine-tuning.

Showed BERT was significantly undertrained; by optimizing training procedure (more data, longer training, no NSP), achieved much better results.

175B-parameter GPT. Pioneered few-shot and in-context learning, dramatically reducing the need for fine-tuning.

Applied the Transformer architecture directly to images by splitting them into patches (16x16 pixel squares) and treating each patch like a word token, eliminating the need for convolutional neural networks.

Trained a model to understand both images and text by learning which image-text pairs go together from 400 million internet examples. This created a shared 'embedding space' where images and text can be directly compared.

Trained a speech recognition model on 680,000 hours of multilingual audio from the internet, achieving near-human accuracy across 97 languages without any task-specific fine-tuning.

Introduced sparsely-gated Mixture-of-Experts layers for scaling model capacity without proportional compute increase.

Pioneered efficient model parallelism techniques enabling training of multi-billion parameter Transformers across GPUs.

Found that model performance follows power laws in compute, parameters, and data. Provided the mathematical framework for scaling decisions.

Simplified MoE routing to scale to trillions of parameters efficiently. Influenced Mixtral and GPT-4/5 MoE architectures.

Proposed freezing the original model weights and injecting small trainable low-rank matrices, reducing fine-tuning memory by 10,000x while maintaining quality.

Challenged Kaplan's scaling laws by showing data should scale equally to parameters. 70B Chinchilla outperformed 280B Gopher.

Showed that smaller models trained on significantly more data (following Chinchilla scaling laws) could match or exceed the performance of much larger models, and released the weights openly.

Described GPT-4's multimodal capabilities and performance across professional/academic benchmarks, setting new SOTA on bar exam, MMLU, and many others.

Provided the most detailed public documentation of how to train, fine-tune, and safety-align a large language model, including their full RLHF methodology.

Introduced sliding window attention and demonstrated that a 7B model could outperform LLaMA 2 13B on all benchmarks.

Introduced the Gemini family with native multimodal training from the ground up, achieving SOTA on 30+ benchmarks.

Showed that SwiGLU activation (Swish + Gated Linear Unit) significantly improves Transformer FFN quality with minimal compute overhead.

Scaled Mixture-of-Experts to 600 billion parameters with automatic model parallelism across thousands of TPUs, showing how to train models far beyond single-device memory limits.

Introduced rotary position embeddings that encode position via rotation matrices, enabling better length generalization. Used by virtually every modern LLM.

Restructured how attention computation accesses GPU memory (tiling and recomputation), achieving 2-4x speedup and enabling much longer context windows without approximation.

Pioneered post-training quantization to 4-bit for large language models with minimal quality loss, enabling consumer GPU inference.

Introduced grouped-query attention as a middle ground between multi-head and multi-query attention, reducing KV cache memory while maintaining quality.

Introduced selective state space models that process sequences in linear time (vs. quadratic for Transformers), with a data-dependent selection mechanism that lets the model focus on relevant parts of the input.

Introduced Multi-head Latent Attention (MLA), which compresses the key-value cache into a low-rank latent space, dramatically reducing the memory needed to serve long-context models.

Showed that gradually adding noise to data and then learning to reverse the process could generate images rivaling GANs, with more stable training and better diversity.

Demonstrated that a single model could generate diverse, creative images from arbitrary text descriptions, combining language understanding with image generation.

Demonstrated that large frozen text encoders (T5-XXL) with cascaded diffusion models produce photorealistic images, outperforming DALL·E 2.

Pioneered the RLHF paradigm — training a reward model from human preferences, then using it to fine-tune policies via reinforcement learning.

Introduced RL from AI Feedback using "constitutions" (rule sets) for self-supervision, reducing reliance on human labels for harmlessness training.

Applied RLHF to GPT-3: supervised fine-tuning → reward modeling → PPO optimization. Made models safer, more helpful, and more aligned.

Showed that preference learning could be formulated as a simple classification problem on pairs of outputs, eliminating the need for a separate reward model and the instabilities of PPO training.

Combined a neural retriever (that searches a knowledge base) with a sequence-to-sequence generator, allowing the model to 'look up' relevant documents before answering — reducing hallucinations and enabling knowledge updates without retraining.

Showed that prompting models to "think step-by-step" unlocks arithmetic, logic, and commonsense reasoning in large models like PaLM.

Combined chain-of-thought reasoning with external tool use (APIs, search), improving QA and decision-making through interleaved reasoning and action.

Demonstrated that pure RL training (without supervised fine-tuning on reasoning traces) can produce chain-of-thought reasoning, achieving performance comparable to OpenAI o1.