mlsanityData to 7 Oct 2026

Language Models are Few-Shot Learners

arXiv 2005.14165 ↗First posted 28 May 2020cs.CLcited by 32,199

On the map: World models → World models for language agents

Abstract (arXiv)

Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task. While typically task-agnostic in architecture, this method still requires task-specific fine-tuning datasets of thousands or tens of thousands of examples. By contrast, humans can generally perform a new language task from only a few examples or from simple instructions - something which current NLP systems still largely struggle to do. Here we show that scaling up language models greatly improves task-agnostic, few-shot performance, sometimes even reaching competitiveness with prior state-of-the-art fine-tuning approaches. Specifically, we train GPT-3, an autoregressive language model with 175 billion parameters, 10x more than any previous non-sparse language model, and test its performance in the few-shot setting. For all tasks, GPT-3 is applied without any gradient updates or fine-tuning, with tasks and few-shot demonstrations specified purely via text interaction with the model. GPT-3 achieves strong performance on many NLP datasets, including translation, question-answering, and cloze tasks, as well as several tasks that require on-the-fly reasoning or domain adaptation, such as unscrambling words, using a novel word in a sentence, or performing 3-digit arithmetic. At the same time, we also identify some datasets where GPT-3's few-shot learning still struggles, as well as some datasets where GPT-3 faces methodological issues related to training on large web corpora. Finally, we find that GPT-3 can generate samples of news articles which human evaluators have difficulty distinguishing from articles written by humans. We discuss broader societal impacts of this finding and of GPT-3 in general.

Key facts. Cited by 32,199 papers in the map (data through 7 Oct 2026). Placed in World models for language agents, one of 11 directions in World models. Builds on 0 of its 139 references that are in the map. First posted on arXiv 28 May 2020.

Builds on

0 of 139 references.

No references here yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 9; 2021: 1727; 2022: 3024; 2023: 7393; 2024: 8879; 2025: 7150; 2026: 4017.

32,199 papers.

Sort:

By area:

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  8. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  9. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  10. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  11. 2025Qwen3 Technical Reportcited by 8,195
  12. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
38 more
  1. 2023Segment Anythingcited by 8,158
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  6. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  7. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  8. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  9. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  10. 2022Scalable Diffusion Models with Transformerscited by 5,931
  11. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  12. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  13. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  14. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  15. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  16. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  17. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  18. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  19. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  20. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  21. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  22. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  23. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  24. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  25. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  26. 2021Program Synthesis with Large Language Modelscited by 3,509
  27. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  28. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  29. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  30. 2023Qwen Technical Reportcited by 3,273
  31. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  32. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  33. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  34. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  35. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  36. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  37. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  38. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689

50 most cited of 32,199 All 32,199 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Optimising Language Models for Downstream Tasks: A Post-Training Perspective
  2. 2024Observational Scaling Laws and the Predictability of Language Model Performance
  3. 2026CLaS-Bench: A Cross-Lingual Alignment and Steering Benchmark
  4. 2025Investigating Test-Time Scaling with Reranking for Machine Translation
  5. 2021Self-training with Few-shot Rationalization: Teacher Explanations Aid Student in Few-shot NLU
  6. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies
  7. 2026Benchmark Shadows: Data Alignment, Parameter Footprints, and Generalization in Large Language Models
  8. 2024Optimizing Large Language Models for Turkish: New Methodologies in Corpus Selection and Training