mlsanityData to 7 Oct 2026

A Theory for Emergence of Complex Skills in Language Models

arXiv 2307.15936 ↗First posted 29 Jul 2023cs.LG cs.AI cs.CLcited by 111

Not on the map. Why?

Abstract (arXiv)

A major driver of AI products today is the fact that new skills emerge in language models when their parameter set and training corpora are scaled up. This phenomenon is poorly understood, and a mechanistic explanation via mathematical analysis of gradient-based training seems difficult. The current paper takes a different approach, analysing emergence using the famous (and empirical) Scaling Laws of LLMs and a simple statistical framework. Contributions include: (a) A statistical framework that relates cross-entropy loss of LLMs to competence on the basic skills that underlie language tasks. (b) Mathematical analysis showing that the Scaling Laws imply a strong form of inductive bias that allows the pre-trained model to learn very efficiently. We informally call this em slingshot generalization since naively viewed it appears to give competence levels at skills that violate usual generalization theory. (c) A key example of slingshot generalization, that competence at executing tasks involving k-tuples of skills emerges essentially at the same scaling and same rate as competence on the elementary skills themselves.

Key facts. Cited by 111 papers in the map (data through 7 Oct 2026). Builds on 13 of its 27 references that are in the map. First posted on arXiv 29 Jul 2023.

Builds on

13 of 27 references.

References · 13

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2020Scaling Laws for Neural Language Modelscited by 6,411
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  5. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  6. 2022Emergent Abilities of Large Language Modelscited by 2,483
  7. 2023PaLM 2 Technical Reportcited by 1,154
  8. 2017Deep Learning Scaling is Predictable, Empiricallycited by 556
  9. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 515
  10. 2021Explaining Neural Scaling Lawscited by 353
  11. 2022Predictability and Surprise in Large Generative Modelscited by 209
  12. 2022Training Trajectories of Language Models Across Scalescited by 80
1 more
  1. 2020A Mathematical Exploration of Why Language Models Help Solve Downstream Taskscited by 71

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 12; 2024: 53; 2025: 28; 2026: 18.

111 papers.

Sort:

By area:

  1. 2024QuRating: Selecting High-Quality Data for Training Language Modelscited by 153
  2. 2024Observational Scaling Laws and the Predictability of Language Model Performancecited by 128
  3. 2024A Tale of Tails: Model Collapse as a Change of Scaling Lawscited by 127
  4. 2024A Dynamical Model of Neural Scaling Lawscited by 105
  5. 2023Compositional Abilities Emerge Multiplicatively: Exploring Diffusion Models on a Synthetic Taskcited by 91
  6. 2024Generalization v.s. Memorization: Tracing Language Models' Capabilities Back to Pretraining Datacited by 83
  7. 2024Metacognitive Capabilities of LLMs: An Exploration in Mathematical Problem Solvingcited by 82
  8. 2024Wisdom of the Silicon Crowd: LLM Ensemble Prediction Capabilities Rival Human Crowd Accuracycited by 66
  9. 2023Skill-Mix: a Flexible and Expandable Family of Evaluations for AI modelscited by 64
  10. 2025The Best Instruction-Tuning Data are Those That Fitcited by 48
  11. 2024Scaling and renormalization in high-dimensional regressioncited by 45
  12. 2024Why Larger Language Models Do In-context Learning Differently?cited by 44
38 more
  1. 2024Theoretical limitations of multi-layer Transformercited by 43
  2. 2024Learning to grok: Emergence of in-context learning and skill composition in modular arithmetic taskscited by 40
  3. 2024Do Large Language Models Have Compositional Ability? An Investigation into Limitations and Scalabilitycited by 39
  4. 2024Towards a theory of how the structure of language is acquired by deep neural networkscited by 38
  5. 2024AI-Assisted Generation of Difficult Math Questionscited by 37
  6. 2025Emergent Abilities in Large Language Models: A Surveycited by 34
  7. 2024Iteration Head: A Mechanistic Study of Chain-of-Thoughtcited by 31
  8. 2024A phase transition between positional and semantic learning in a solvable model of dot-product attentioncited by 30
  9. 2024Out-of-distribution generalization via composition: a lens through induction heads in Transformerscited by 28
  10. 2024Gradient Routing: Masking Gradients to Localize Computation in Neural Networkscited by 28
  11. 2023How to Protect Copyright Data in Optimization of Large Language Models?cited by 26
  12. 2024Emergence in non-neural models: grokking modular arithmetic via average gradient outer productcited by 26
  13. 2024A Percolation Model of Emergence: Analyzing Transformers Trained on a Formal Languagecited by 26
  14. 2025The emergence of sparse attention: impact of data distribution and benefits of repetitioncited by 25
  15. 2023Large Language Model Prediction Capabilities: Evidence from a Real-World Forecasting Tournamentcited by 24
  16. 2024MERA: A Comprehensive LLM Evaluation in Russiancited by 22
  17. 2024On Provable Length and Compositional Generalizationcited by 22
  18. 2024Multilingual Blending: LLM Safety Alignment Evaluation with Language Mixturecited by 22
  19. 2023Compositional Capabilities of Autoregressive Transformers: A Study on Synthetic, Interpretable Taskscited by 21
  20. 2024AI-Augmented Predictions: LLM Assistants Improve Human Forecasting Accuracycited by 21
  21. 2024Advancing Social Intelligence in AI Agents: Technical Challenges and Open Questionscited by 21
  22. 2024Bypassing the Exponential Dependency: Looped Transformers Efficiently Learn In-context by Multi-step Gradient Descentcited by 21
  23. 2024On the Inductive Bias of Stacking Towards Improving Reasoningcited by 18
  24. 2024Can Models Learn Skill Composition from Examples?cited by 17
  25. 2024AcTracer: Active Testing of Large Language Model via Multi-Stage Samplingcited by 16
  26. 2024A Survey on Compositional Learning of AI Models: Theoretical and Experimental Practicescited by 15
  27. 2025Recitation over Reasoning: How Cutting-Edge Language Models Can Fail on Elementary School-Level Reasoning Problems?cited by 15
  28. 2025Hidden Breakthroughs in Language Model Trainingcited by 15
  29. 2024Neural Scaling Laws Rooted in the Data Distributioncited by 14
  30. 2025Context parroting: A simple but tough-to-beat baseline for foundation models in scientific machine learningcited by 14
  31. 2024Learning sum of diverse features: computational hardness and efficient gradient-based training for ridge combinationscited by 13
  32. 2025Does Data Scaling Lead to Visual Compositional Generalization?cited by 13
  33. 2024A Survey to Recent Progress Towards Understanding In-Context Learningcited by 12
  34. 2023Unmasking Transformers: A Theoretical Approach to Data Recovery via Attention Weightscited by 11
  35. 2025Compute Optimal Scaling of Skills: Knowledge vs Reasoningcited by 11
  36. 2024Predicting Emergent Capabilities by Finetuningcited by 10
  37. 2023Local Convergence of Approximate Newton Method for Two Layer Nonlinear Regressioncited by 9
  38. 2024Learning Beyond Pattern Matching? Assaying Mathematical Understanding in LLMscited by 9

50 most cited of 111 All 111 (JSON)