mlsanityData to 7 Oct 2026

Predictability and Surprise in Large Generative Models

arXiv 2202.07785 ↗First posted 15 Feb 2022cs.CYcited by 209

Not on the map. Why?

Abstract (arXiv)

Large-scale pre-training has recently emerged as a technique for creating capable, general purpose, generative models such as GPT-3, Megatron-Turing NLG, Gopher, and many others. In this paper, we highlight a counterintuitive property of such models and discuss the policy implications of this property. Namely, these generative models have an unusual combination of predictable loss on a broad training distribution (as embodied in their "scaling laws"), and unpredictable specific capabilities, inputs, and outputs. We believe that the high-level predictability and appearance of useful capabilities drives rapid development of such models, while the unpredictable qualities make it difficult to anticipate the consequences of model deployment. We go through examples of how this combination can lead to socially harmful behavior with examples from the literature and real world observations, and we also perform two novel experiments to illustrate our point about harms from unpredictability. Furthermore, we analyze how these conflicting properties combine to give model developers various motivations for deploying these models, and challenges that can hinder deployment. We conclude with a list of possible interventions the AI community may take to increase the chance of these models having a beneficial impact. We intend this paper to be useful to policymakers who want to understand and regulate AI systems, technologists who care about the potential policy impact of their work, and academics who want to analyze, critique, and potentially develop large generative models.

Key facts. Cited by 209 papers in the map (data through 7 Oct 2026). First posted on arXiv 15 Feb 2022.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 25; 2023: 69; 2024: 59; 2025: 38; 2026: 18.

209 papers.

Sort:

By area:

  1. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  2. 2022Emergent Abilities of Large Language Modelscited by 2,483
  3. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  4. 2023Progress measures for grokking via mechanistic interpretabilitycited by 822
  5. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  6. 2023Towards Measuring the Representation of Subjective Global Opinions in Language Modelscited by 424
  7. 2023Larger language models do in-context learning differentlycited by 386
  8. 2023Visual Adversarial Examples Jailbreak Aligned Large Language Modelscited by 307
  9. 2023Are aligned neural networks adversarially aligned?cited by 305
  10. 2023Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attackscited by 268
  11. 2022Can language models learn from explanations in context?cited by 248
  12. 2022The Alignment Problem from a Deep Learning Perspectivecited by 246
38 more
  1. 2022Towards Understanding Grokking: An Effective Theory of Representation Learningcited by 229
  2. 2023Inverse Scaling: When Bigger Isn't Bettercited by 205
  3. 2024ArtPrompt: ASCII Art-based Jailbreak Attacks against Aligned LLMscited by 197
  4. 2025Large Language Model Agent: A Survey on Methodology, Applications and Challengescited by 194
  5. 2022Language models show human-like content effects on reasoning taskscited by 171
  6. 2023The Capacity for Moral Self-Correction in Large Language Modelscited by 171
  7. 2023Model evaluation for extreme riskscited by 167
  8. 2023Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learningcited by 155
  9. 2022Instruction Induction: From Few Examples to Natural Language Task Descriptionscited by 152
  10. 2022Scaling Laws and Interpretability of Learning from Repeated Datacited by 147
  11. 2023Emergent and Predictable Memorization in Large Language Modelscited by 147
  12. 2023A Toy Model of Universality: Reverse Engineering How Networks Learn Group Operationscited by 141
  13. 2023The Quantization Model of Neural Scalingcited by 137
  14. 2023Auditing large language models: a three-layered approachcited by 135
  15. 2023Harms from Increasingly Agentic Algorithmic Systemscited by 134
  16. 2023Taken out of context: On measuring situational awareness in LLMscited by 131
  17. 2023Self-Taught Optimizer (STOP): Recursively Self-Improving Code Generationcited by 128
  18. 2023AI Transparency in the Age of LLMs: A Human-Centered Research Roadmapcited by 119
  19. 2023Foundation Models and Fair Usecited by 112
  20. 2023A Theory for Emergence of Complex Skills in Language Modelscited by 111
  21. 2022Broken Neural Scaling Lawscited by 102
  22. 2023Regulating ChatGPT and other Large Generative AI Modelscited by 102
  23. 2023The Curious Decline of Linguistic Diversity: Training Language Models on Synthetic Textcited by 97
  24. 2023Frontier AI Regulation: Managing Emerging Risks to Public Safetycited by 95
  25. 2024Understanding Emergent Abilities of Language Models from the Loss Perspectivecited by 94
  26. 2024Revisiting Catastrophic Forgetting in Large Language Model Tuningcited by 93
  27. 2022What Language Model to Train if You Have One Million GPU Hours?cited by 92
  28. 2023Language Model Behavior: A Comprehensive Surveycited by 92
  29. 2022Training Trajectories of Language Models Across Scalescited by 80
  30. 2023Cognitive Overload: Jailbreaking Large Language Models with Overloaded Logical Thinkingcited by 78
  31. 2024Visibility into AI Agentscited by 77
  32. 2022Inverse scaling can become U-shapedcited by 76
  33. 2023A Benchmark for Learning to Translate a New Language from One Grammar Bookcited by 74
  34. 2023Eight Things to Know about Large Language Modelscited by 73
  35. 2024The Multilingual Alignment Prism: Aligning Global and Local Preferences to Reduce Harmcited by 67
  36. 2024Diffusion Models, Image Super-Resolution And Everything: A Surveycited by 63
  37. 2022X-Risk Analysis for AI Researchcited by 60
  38. 2022Transcending Scaling Laws with 0.1% Extra Computecited by 55

50 most cited of 209 All 209 (JSON)