mlsanityData to 14 Oct 2026

Risks from Learned Optimization in Advanced Machine Learning Systems

arXiv 1906.01820 ↗First posted 5 Jun 2019cs.AIcited by 258

Not on the map. Why?

Cited by 35 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We analyze the type of learned optimization that occurs when a learned model (such as a neural network) is itself an optimizer - a situation we refer to as mesa-optimization, a neologism we introduce in this paper. We believe that the possibility of mesa-optimization raises two important questions for the safety and transparency of advanced machine learning systems. First, under what circumstances will learned models be optimizers, including when they should not be? Second, when a learned model is an optimizer, what will its objective be - how will it differ from the loss function it was trained under - and how can it be aligned? In this paper, we provide an in-depth analysis of these two primary questions and provide an overview of topics for future research.

Key facts. Cited by 258 papers in the map (data through 14 Oct 2026). First posted on arXiv 5 Jun 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 5; 2022: 22; 2023: 24; 2024: 48; 2025: 67; 2026: 92.

258 papers.

Sort:

By area:

  1. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  2. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 836
  3. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 832
  4. 2022Transformers learn in-context by gradient descentcited by 732
  5. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 713
  6. 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Trainingcited by 529
  7. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 467
  8. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 422
  9. 2024Alignment faking in large language modelscited by 360
  10. 2025Absolute Zero: Reinforced Self-play Reasoning with Zero Datacited by 344
  11. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 328
  12. 2021Unsolved Problems in ML Safetycited by 292
38 more
  1. 2024Frontier Models are Capable of In-context Schemingcited by 287
  2. 2022The Alignment Problem from a Deep Learning Perspectivecited by 258
  3. 2023Large Language Model Alignment: A Surveycited by 243
  4. 2022Measuring Progress on Scalable Oversight for Large Language Modelscited by 225
  5. 2023Inverse Scaling: When Bigger Isn't Bettercited by 207
  6. 2024Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Modelscited by 178
  7. 2021Alignment of Language Agentscited by 168
  8. 2023Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchillacited by 149
  9. 2023Pretraining task diversity and the emergence of non-Bayesian in-context learning for regressioncited by 146
  10. 2024AI Sandbagging: Language Models can Strategically Underperform on Evaluationscited by 145
  11. 2023Taken out of context: On measuring situational awareness in LLMscited by 143
  12. 2023Getting aligned on representational alignmentcited by 143
  13. 2025Natural Emergent Misalignment from Reward Hacking in Production RLcited by 136
  14. 2022Goal Misgeneralization: Why Correct Specifications Aren't Enough For Correct Goalscited by 126
  15. 2021Goal Misgeneralization in Deep Reinforcement Learningcited by 123
  16. 2024Secret Collusion among AI Agents: Multi-Agent Deception via Steganographycited by 113
  17. 2022Is Power-Seeking AI an Existential Risk?cited by 105
  18. 2023Deception Abilities Emerged in Large Language Modelscited by 101
  19. 2023Uncovering mesa-optimization algorithms in Transformerscited by 100
  20. 2024On scalable oversight with weak LLMs judging strong LLMscited by 99
  21. 2023Frontier AI Regulation: Managing Emerging Risks to Public Safetycited by 98
  22. 2024Towards Guaranteed Safe AI: A Framework for Ensuring Robust and Reliable AI Systemscited by 96
  23. 2023Still No Lie Detector for Language Models: Probing Empirical and Conceptual Roadblockscited by 78
  24. 2025Test-time regression: a unifying framework for designing sequence models with associative memorycited by 74
  25. 2023Eight Things to Know about Large Language Modelscited by 73
  26. 2023Eliciting Latent Knowledge from Quirky Language Modelscited by 62
  27. 2022Adversarial Training for High-Stakes Reliabilitycited by 60
  28. 2022X-Risk Analysis for AI Researchcited by 60
  29. 2024Sabotage Evaluations for Frontier Modelscited by 52
  30. 2024Beyond Preferences in AI Alignmentcited by 49
  31. 2025MesaNet: Sequence Modeling by Locally Optimal Test-Time Trainingcited by 47
  32. 2025Learning without training: The implicit dynamics of in-context learningcited by 43
  33. 2022A Modern Self-Referential Weight Matrix That Learns to Modify Itselfcited by 41
  34. 2023Honesty Is the Best Policy: Defining and Mitigating AI Deceptioncited by 40
  35. 2021The Dangers of Underclaiming: Reasons for Caution When Reporting How NLP Systems Failcited by 38
  36. 2024Uncovering Deceptive Tendencies in Language Models: A Simulated Company AI Assistantcited by 38
  37. 2025Evaluating Frontier Models for Stealth and Situational Awarenesscited by 37
  38. 2024Personality Alignment of Large Language Modelscited by 36

50 most cited of 258 All 258 (JSON)