mlsanityData to 7 Oct 2026

Are Emergent Abilities in Large Language Models just In-Context Learning?

arXiv 2309.01809 ↗First posted 4 Sep 2023cs.CLcited by 109

Not on the map. Why?

Abstract (arXiv)

Large language models, comprising billions of parameters and pre-trained on extensive web-scale corpora, have been claimed to acquire certain capabilities without having been specifically trained on them. These capabilities, referred to as "emergent abilities," have been a driving force in discussions regarding the potentials and risks of language models. A key challenge in evaluating emergent abilities is that they are confounded by model competencies that arise through alternative prompting techniques, including in-context learning, which is the ability of models to complete a task based on a few examples. We present a novel theory that explains emergent abilities, taking into account their potential confounding factors, and rigorously substantiate this theory through over 1000 experiments. Our findings suggest that purported emergent abilities are not truly emergent, but result from a combination of in-context learning, model memory, and linguistic knowledge. Our work is a foundational step in explaining language model performance, providing a template for their efficient use and clarifying the paradox of their ability to excel in some instances while faltering in others. Thus, we demonstrate that their capabilities should not be overestimated.

Key facts. Cited by 109 papers in the map (data through 7 Oct 2026). Builds on 57 of its 82 references that are in the map. First posted on arXiv 4 Sep 2023.

Builds on

57 of 82 references.

References · 57

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  8. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  9. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  10. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  11. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  12. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
38 more
  1. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  2. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  3. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  4. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  5. 2022Emergent Abilities of Large Language Modelscited by 2,483
  6. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  7. 2019Language Models as Knowledge Bases?cited by 1,922
  8. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  9. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  10. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  11. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  12. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  13. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  14. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  15. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 888
  16. 2020A Primer in BERTology: What we know about how BERT workscited by 807
  17. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
  18. 2022Transformers learn in-context by gradient descentcited by 714
  19. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  20. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 515
  21. 2018What you can cram into a single vector: Probing sentence embeddings for linguistic propertiescited by 452
  22. 2019What do you learn from context? Probing for sentence structure in contextualized word representationscited by 436
  23. 2023Larger language models do in-context learning differentlycited by 386
  24. 2023Trained Transformers Learn Linear Models In-Contextcited by 331
  25. 2019What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language modelscited by 317
  26. 2022Data Distributional Properties Drive Emergent In-Context Learning in Transformerscited by 303
  27. 2022Can language models learn from explanations in context?cited by 248
  28. 2023Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Taskscited by 243
  29. 2020Information-Theoretic Probing with Minimum Description Lengthcited by 230
  30. 2023Transformers as Algorithms: Generalization and Stability in In-context Learningcited by 220
  31. 2022Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizerscited by 204
  32. 2021Reframing Instructional Prompts to GPTk's Languagecited by 176
  33. 2023Fundamental Limitations of Alignment in Large Language Modelscited by 152
  34. 2023Emergent and Predictable Memorization in Large Language Modelscited by 147
  35. 2023The Learnability of In-Context Learningcited by 126
  36. 2019Open Sesame: Getting Inside BERT's Linguistic Knowledgecited by 119
  37. 2023What and How does In-Context Learning Learn? Bayesian Model Averaging, Parameterization, and Generalizationcited by 112
  38. 2020When Do You Need Billions of Words of Pretraining Data?cited by 104

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 12; 2024: 52; 2025: 29; 2026: 16.

109 papers.

Sort:

By area:

  1. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrationscited by 401
  2. 2024Observational Scaling Laws and the Predictability of Language Model Performancecited by 128
  3. 2023Factuality Challenges in the Era of Large Language Modelscited by 125
  4. 2024Visual In-Context Learning for Large Vision-Language Modelscited by 124
  5. 2023The mechanistic basis of data dependence and abrupt learning in an in-context classification taskcited by 115
  6. 2024Personalized Pieces: Efficient Personalized Large Language Models through Collaborative Effortscited by 71
  7. 2023From Language Modeling to Instruction Following: Understanding the Behavior Shift in LLMs after Instruction Tuningcited by 56
  8. 2024What Makes Multimodal In-Context Learning Work?cited by 54
  9. 2024A Comprehensive Evaluation of Quantization Strategies for Large Language Modelscited by 47
  10. 2024ALI-Agent: Assessing LLMs' Alignment with Human Values via Agent-based Evaluationcited by 46
  11. 2024A Theoretical Understanding of Self-Correction through In-context Alignmentcited by 43
  12. 2024In-Context Principle Learning from Mistakescited by 42
38 more
  1. 2024Compositional Generative Modeling: A Single Model is Not All You Needcited by 41
  2. 2024Detecting Hallucinations in Large Language Model Generation: A Token Probability Approachcited by 40
  3. 2023Retrieval-augmented Generation to Improve Math Question-Answering: Trade-offs Between Groundedness and Human Preferencecited by 37
  4. 2023The Mystery of In-Context Learning: A Comprehensive Survey on Interpretation and Analysiscited by 34
  5. 2025Emergent Abilities in Large Language Models: A Surveycited by 34
  6. 2024Training on the Test Task Confounds Evaluation and Emergencecited by 33
  7. 2024Parallel Structures in Pre-training Data Yield In-Context Learningcited by 26
  8. 2024Holmes: A Benchmark to Assess the Linguistic Competence of Language Modelscited by 26
  9. 2024A Percolation Model of Emergence: Analyzing Transformers Trained on a Formal Languagecited by 26
  10. 2023Competition-Level Problems are Effective LLM Evaluatorscited by 25
  11. 2024Delving into Multi-modal Multi-task Foundation Models for Road Scene Understanding: From Learning Paradigm Perspectivescited by 22
  12. 2024On Large Language Models in National Security Applicationscited by 18
  13. 2024Quantifying the Capabilities of LLMs across Scale and Precisioncited by 17
  14. 2024Language Models are Surprisingly Fragile to Drug Names in Biomedical Benchmarkscited by 17
  15. 2023Why Can Large Language Models Generate Correct Chain-of-Thoughts?cited by 16
  16. 2024TinyStyler: Efficient Few-Shot Text Style Transfer with Authorship Embeddingscited by 16
  17. 2024A Survey on Compositional Learning of AI Models: Theoretical and Experimental Practicescited by 15
  18. 2024Logical Consistency of Large Language Models in Fact-checkingcited by 15
  19. 2023In-Context Learning Dynamics with Random Binary Sequencescited by 13
  20. 2024Exploring the Robustness of In-Context Learning with Noisy Labelscited by 13
  21. 2025LazyReview A Dataset for Uncovering Lazy Thinking in NLP Peer Reviewscited by 13
  22. 2024MLPs Learn In-Context on Regression and Classification Taskscited by 12
  23. 2024Does learning the right latent variables necessarily improve in-context learning?cited by 12
  24. 2025Triangulating LLM Progress through Benchmarks, Games, and Cognitive Testscited by 11
  25. 2023Language Models as a Service: Overview of a New Paradigm and its Challengescited by 10
  26. 2024AI-enhanced Collective Intelligencecited by 10
  27. 2024Predicting Emergent Capabilities by Finetuningcited by 10
  28. 2024Memory, Consciousness and Large Language Modelcited by 9
  29. 2024An Empirical Study of In-context Learning in LLMs for Machine Translationcited by 9
  30. 2024Towards an Understanding of Stepwise Inference in Transformers: A Synthetic Graph Navigation Modelcited by 9
  31. 2024Assessing the Reasoning Capabilities of LLMs in the context of Evidence-based Claim Verificationcited by 9
  32. 2023Breaking through the learning plateaus of in-context learning in Transformercited by 8
  33. 2024A Multi-Perspective Analysis of Memorization in Large Language Modelscited by 8
  34. 2024U-shaped and Inverted-U Scaling behind Emergent Abilities of Large Language Modelscited by 8
  35. 2025Large Language Models and Emergence: A Complex Systems Perspectivecited by 7
  36. 2024FsPONER: Few-shot Prompt Optimization for Named Entity Recognition in Domain-specific Scenarioscited by 6
  37. 2024Different Tastes of Entities: Investigating Human Label Variation in Named Entity Annotationscited by 5
  38. 2024M2QA: Multi-domain Multilingual Question Answeringcited by 5

50 most cited of 109 All 109 (JSON)