mlsanityData to 14 Oct 2026

A Latent Space Theory for Emergent Abilities in Large Language Models

arXiv 2304.09960 ↗First posted 19 Apr 2023cs.CL cs.AI cs.LGcited by 43

Not on the map. Why?

Abstract (arXiv)

Languages are not created randomly but rather to communicate information. There is a strong association between languages and their underlying meanings, resulting in a sparse joint distribution that is heavily peaked according to their correlations. Moreover, these peak values happen to match with the marginal distribution of languages due to the sparsity. With the advent of LLMs trained on big data and large models, we can now precisely assess the marginal distribution of languages, providing a convenient means of exploring the sparse structures in the joint distribution for effective inferences. In this paper, we categorize languages as either unambiguous or ε-ambiguous and present quantitative results to demonstrate that the emergent abilities of LLMs, such as language understanding, in-context learning, chain-of-thought prompting, and effective instruction fine-tuning, can all be attributed to Bayesian inference on the sparse joint distribution of languages.

Key facts. Cited by 43 papers in the map (data through 14 Oct 2026). Builds on 22 of its 31 references that are in the map. First posted on arXiv 19 Apr 2023.

Builds on

22 of 31 references.

References · 22

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  6. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  7. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  8. 2022Emergent Abilities of Large Language Modelscited by 2,512
  9. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  10. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  11. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,195
  12. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 938
10 more
  1. 2021Learning To Retrieve Prompts for In-Context Learningcited by 648
  2. 2019Are Transformers universal approximators of sequence-to-sequence functions?cited by 406
  3. 2022Data Distributional Properties Drive Emergent In-Context Learning in Transformerscited by 309
  4. 2022Impact of Pretraining Term Frequencies on Few-Shot Reasoningcited by 142
  5. 2023The Learnability of In-Context Learningcited by 127
  6. 2023A Theory of Emergent In-Context Learning as Implicit Structure Inductioncited by 86
  7. 2021Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuningcited by 82
  8. 2022Entailment Semantics Can Be Extracted from an Ideal Language Modelcited by 10
  9. 2020On Approximation Capabilities of ReLU Activation and Softmax Output Layer in Neural Networkscited by 5
  10. 2019Why Learning of Large-Scale Neural Networks Behaves Like Convex Optimizationcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 12; 2024: 13; 2025: 11; 2026: 7.

43 papers.

Sort:

By area:

  1. 2023What and How does In-Context Learning Learn? Bayesian Model Averaging, Parameterization, and Generalizationcited by 118
  2. 2023In-Context Convergence of Transformerscited by 115
  3. 2023Are Emergent Abilities in Large Language Models just In-Context Learning?cited by 109
  4. 2023Still No Lie Detector for Language Models: Probing Empirical and Conceptual Roadblockscited by 78
  5. 2024Demystifying Chains, Trees, and Graphs of Thoughtscited by 63
  6. 2024Is In-Context Learning in Large Language Models Bayesian? A Martingale Perspectivecited by 57
  7. 2023In-Context Learning Learns Label Relationships but Is Not Conventional Learningcited by 51
  8. 2023Reason for Future, Act for Now: A Principled Framework for Autonomous LLM Agents with Provable Sample Efficiencycited by 46
  9. 2024In-Context Learning with Representations: Contextual Generalization of Trained Transformerscited by 38
  10. 2024Do LLM Agents Have Regret? A Case Study in Online Learning and Gamescited by 35
  11. 2024Efficient Reinforcement Learning with Large Language Model Priorscited by 30
  12. 2024Unveiling the Statistical Foundations of Chain-of-Thought Prompting Methodscited by 29
31 more
  1. 2024Mind the Inconspicuous: Revealing the Hidden Weakness in Aligned LLMs' Refusal Boundariescited by 28
  2. 2025Multi-Agent Debate for LLM Judges with Adaptive Stability Detectioncited by 23
  3. 2023BookGPT: A General Framework for Book Recommendation Empowered by Large Language Modelcited by 22
  4. 2024Estimating the Hallucination Rate of Generative AIcited by 20
  5. 2024Bayesian scaling laws for in-context learningcited by 19
  6. 2023Trainable Transformer in Transformercited by 17
  7. 2024From Words to Actions: Unveiling the Theoretical Underpinnings of LLM-Driven Autonomous Systemscited by 17
  8. 2023Why Can Large Language Models Generate Correct Chain-of-Thoughts?cited by 16
  9. 2026Training Language Models via Neural Cellular Automatacited by 15
  10. 2025Video Latent Flow Matching: Optimal Polynomial Projections for Video Interpolation and Extrapolationcited by 11
  11. 2023Towards Semantic Communication Protocols for 6G: From Protocol Learning to Language-Oriented Approachescited by 7
  12. 2025The Role of Diversity in In-Context Learning for Large Language Modelscited by 6
  13. 2025JAM: Controllable and Responsible Text Generation via Causal Reasoning and Latent Vector Manipulationcited by 5
  14. 2023Amplifying Limitations, Harms and Risks of Large Language Modelscited by 4
  15. 2025Why is prompting hard? Understanding prompts on binary sequence predictorscited by 4
  16. 2024Knowledge Distillation from Language-Oriented to Emergent Communication for Multi-Agent Remote Controlcited by 3
  17. 2025Towards Auto-Regressive Next-Token Prediction: In-Context Learning Emerges from Generalizationcited by 3
  18. 2026From Drift to Coherence: Stabilizing Beliefs in LLMscited by 3
  19. 2023Steering Language Generation: Harnessing Contrastive Expert Guidance and Negative Prompting for Coherent and Diverse Synthetic Data Generationcited by 2
  20. 2025Fine-Grained Emotion Recognition via In-Context Learningcited by 2
  21. 2024Dynamic and Super-Personalized Media Ecosystem Driven by Generative AI: Unpredictable Plays Never Repeating The Samecited by 1
  22. 2025Transformers as Multi-task Learners: Decoupling Features in Hidden Markov Modelscited by 1
  23. 2025On the Relationship Between the Choice of Representation and In-Context Learningcited by 1
  24. 2025Single LLM Debate, MoLaCE: Mixture of Latent Concept Experts Against Confirmation Biascited by 1
  25. 2026Beyond the Prompt in Large Language Models: Comprehension, In-Context Learning, and Chain-of-Thoughtcited by 1
  26. 2026Autoregressive Learning in Joint KL: Sharp Oracle Bounds and Lower Boundscited by 1
  27. 2024MacBehaviour: An R package for behavioural experimentation on large language modelscited by 0
  28. 2025Strategic Prompt Pricing for AIGC Services: A User-Centric Approachcited by 0
  29. 2026Understanding In-Context Multimodal Jailbreaks via Posterior Reweightingcited by 0
  30. 2026Small transformers track Bayesian evidence for latent common causes via a context-invariant mechanismcited by 0
  31. 2026Theory on Attention Dynamics for Out-of-Distribution In-Context Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Emergent Abilities in Large Language Models: A Survey
  2. 2023Do Emergent Abilities Exist in Quantized Large Language Models: An Empirical Study
  3. 2023Large Linguistic Models: Investigating LLMs' metalinguistic abilities
  4. 2026What do Language Models Learn and When? The Implicit Curriculum Hypothesis
  5. 2025The Birth of Knowledge: Emergent Features across Time, Space, and Scale in Large Language Models
  6. 2025Do Large Language Models Know What They Are Capable Of?
  7. 2026UniCog: Uncovering Cognitive Abilities of LLMs through Latent Mind Space Analysis
  8. 2023Is Bigger and Deeper Always Better? Probing LLaMA Across Scales and Layers