mlsanityData to 7 Oct 2026

Are Emergent Abilities of Large Language Models a Mirage?

arXiv 2304.15004 ↗First posted 28 Apr 2023cs.AI cs.LGcited by 515

Not on the map. Why?

Cited by 50 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recent work claims that large language models display emergent abilities, abilities not present in smaller-scale models that are present in larger-scale models. What makes emergent abilities intriguing is two-fold: their sharpness, transitioning seemingly instantaneously from not present to present, and their unpredictability, appearing at seemingly unforeseeable model scales. Here, we present an alternative explanation for emergent abilities: that for a particular task and model family, when analyzing fixed model outputs, emergent abilities appear due to the researcher's choice of metric rather than due to fundamental changes in model behavior with scale. Specifically, nonlinear or discontinuous metrics produce apparent emergent abilities, whereas linear or continuous metrics produce smooth, continuous predictable changes in model performance. We present our alternative explanation in a simple mathematical model, then test it in three complementary ways: we (1) make, test and confirm three predictions on the effect of metric choice using the InstructGPT/GPT-3 family on tasks with claimed emergent abilities; (2) make, test and confirm two predictions about metric choices in a meta-analysis of emergent abilities on BIG-Bench; and (3) show to choose metrics to produce never-before-seen seemingly emergent abilities in multiple vision tasks across diverse deep networks. Via all three analyses, we provide evidence that alleged emergent abilities evaporate with different metrics or with better statistics, and may not be a fundamental property of scaling AI models.

Key facts. Cited by 515 papers in the map (data through 7 Oct 2026). Builds on 24 of its 39 references that are in the map. First posted on arXiv 28 Apr 2023.

Builds on

24 of 39 references.

References · 24

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020Scaling Laws for Neural Language Modelscited by 6,411
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  5. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  6. 2022Emergent Abilities of Large Language Modelscited by 2,483
  7. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  8. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  9. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  10. 2021Scaling Vision Transformerscited by 993
  11. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 599
  12. 2017Deep Learning Scaling is Predictable, Empiricallycited by 556
12 more
  1. 2020Scaling Laws for Autoregressive Generative Modelingcited by 510
  2. 2022Data Distributional Properties Drive Emergent In-Context Learning in Transformerscited by 303
  3. 2021Scaling Laws for Transfercited by 265
  4. 2022Unified Scaling Laws for Routed Language Modelscited by 249
  5. 2019A Constructive Prediction of the Generalization Error Across Scalescited by 202
  6. 2022Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limitcited by 184
  7. 2022Omnigrok: Grokking Beyond Algorithmic Datacited by 142
  8. 2023The Quantization Model of Neural Scalingcited by 137
  9. 2022Broken Neural Scaling Lawscited by 102
  10. 2023Grokking modular arithmeticcited by 77
  11. 2021Scaling Scaling Laws with Board Gamescited by 62
  12. 2022Scaling Laws for a Multi-Agent Reinforcement Learning Modelcited by 37

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 71; 2024: 180; 2025: 132; 2026: 132.

515 papers.

Sort:

By area:

  1. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  2. 2023Large Language Modelscited by 774
  3. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  4. 2024Yi: Open Foundation Models by 01.AIcited by 706
  5. 2024GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Modelscited by 533
  6. 2025Multi-Agent Collaboration Mechanisms: A Survey of LLMscited by 424
  7. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 415
  8. 2024R-Judge: Benchmarking Safety Risk Awareness for LLM Agentscited by 268
  9. 2023A Survey on Model Compression for Large Language Modelscited by 263
  10. 2024Rethinking Machine Unlearning for Large Language Modelscited by 247
  11. 2024Scaling Large Language Model-based Multi-Agent Collaborationcited by 235
  12. 2024The Unreasonable Ineffectiveness of the Deeper Layerscited by 219
38 more
  1. 2023Navigate through Enigmatic Labyrinth A Survey of Chain of Thought Reasoning: Advances, Frontiers and Futurecited by 184
  2. 2024Lessons from the Trenches on Reproducible Evaluation of Language Modelscited by 172
  3. 2024Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Toolscited by 170
  4. 2023Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsitycited by 163
  5. 2023A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethicscited by 162
  6. 2023Sociotechnical Safety Evaluation of Generative AI Systemscited by 154
  7. 2023The Clock and the Pizza: Two Stories in Mechanistic Explanation of Neural Networkscited by 153
  8. 2024The Impact of Reasoning Step Length on Large Language Modelscited by 152
  9. 2023D4: Improving LLM Pretraining via Document De-Duplication and Diversificationcited by 146
  10. 2024Exploring Large Language Model based Intelligent Agents: Definitions, Methods, and Prospectscited by 130
  11. 2023Self-Taught Optimizer (STOP): Recursively Self-Improving Code Generationcited by 128
  12. 2024LLM Agents can Autonomously Exploit One-day Vulnerabilitiescited by 125
  13. 2024Black-Box Access is Insufficient for Rigorous AI Auditscited by 123
  14. 2023Sudden Drops in the Loss: Syntax Acquisition, Phase Transitions, and Simplicity Bias in MLMscited by 121
  15. 2024The Evolution of Statistical Induction Heads: In-Context Learning Markov Chainscited by 115
  16. 2023A Theory for Emergence of Complex Skills in Language Modelscited by 111
  17. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 109
  18. 2023Are Emergent Abilities in Large Language Models just In-Context Learning?cited by 109
  19. 2024Secret Collusion among AI Agents: Multi-Agent Deception via Steganographycited by 107
  20. 2024Language models scale reliably with over-training and on downstream taskscited by 105
  21. 2024Can Mamba Learn How to Learn? A Comparative Study on In-Context Learning Taskscited by 104
  22. 2025Big-Math: A Large-Scale, High-Quality Math Dataset for Reinforcement Learning in Language Modelscited by 104
  23. 2023Grokking as the Transition from Lazy to Rich Training Dynamicscited by 102
  24. 2025Measuring what Matters: Construct Validity in Large Language Model Benchmarkscited by 102
  25. 2023STEVE-1: A Generative Model for Text-to-Behavior in Minecraftcited by 95
  26. 2023Frontier AI Regulation: Managing Emerging Risks to Public Safetycited by 95
  27. 2023Evaluating Language-Model Agents on Realistic Autonomous Taskscited by 94
  28. 2024Understanding Emergent Abilities of Language Models from the Loss Perspectivecited by 94
  29. 2024Learning Dynamics of LLM Finetuningcited by 93
  30. 2023Compositional Abilities Emerge Multiplicatively: Exploring Diffusion Models on a Synthetic Taskcited by 91
  31. 2023T-Eval: Evaluating the Tool Utilization Capability of Large Language Models Step by Stepcited by 91
  32. 2024From Matching to Generation: A Survey on Generative Information Retrievalcited by 89
  33. 2024The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Modelscited by 89
  34. 2024How Do Large Language Models Acquire Factual Knowledge During Pretraining?cited by 86
  35. 2023Goat: Fine-tuned LLaMA Outperforms GPT-4 on Arithmetic Taskscited by 84
  36. 2024Resolving Discrepancies in Compute-Optimal Scaling of Language Modelscited by 84
  37. 2023NPHardEval: Dynamic Benchmark on Reasoning Ability of Large Language Models via Complexity Classescited by 78
  38. 2023A Benchmark for Learning to Translate a New Language from One Grammar Bookcited by 74

50 most cited of 515 All 515 (JSON)