mlsanityData to 7 Oct 2026

Model evaluation for extreme risks

arXiv 2305.15324 ↗First posted 24 May 2023cs.AIcited by 167

Not on the map. Why?

Cited by 20 in LLM alignment / RLHF / preference optimisation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Current approaches to building general-purpose AI systems tend to produce systems with both beneficial and harmful capabilities. Further progress in AI development could lead to capabilities that pose extreme risks, such as offensive cyber capabilities or strong manipulation skills. We explain why model evaluation is critical for addressing extreme risks. Developers must be able to identify dangerous capabilities (through "dangerous capability evaluations") and the propensity of models to apply their capabilities for harm (through "alignment evaluations"). These evaluations will become critical for keeping policymakers and other stakeholders informed, and for making responsible decisions about model training, deployment, and security.

Key facts. Cited by 167 papers in the map (data through 7 Oct 2026). Builds on 37 of its 63 references that are in the map. First posted on arXiv 24 May 2023.

Builds on

37 of 63 references.

References · 37

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  4. 2022Emergent Abilities of Large Language Modelscited by 2,483
  5. 2018Model Cards for Model Reportingcited by 1,068
  6. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  7. 2022Red Teaming Language Models with Language Modelscited by 983
  8. 2023Progress measures for grokking via mechanistic interpretabilitycited by 822
  9. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  10. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 733
  11. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  12. 2020Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditingcited by 380
25 more
  1. 2022The Alignment Problem from a Deep Learning Perspectivecited by 246
  2. 2022Predictability and Surprise in Large Generative Modelscited by 209
  3. 2023Automatically Auditing Large Language Models via Discrete Optimizationcited by 187
  4. 2023Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmarkcited by 174
  5. 2020Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claimscited by 163
  6. 2018The Malicious Use of Artificial Intelligence: Forecasting, Prevention, and Mitigationcited by 159
  7. 2023Auditing large language models: a three-layered approachcited by 135
  8. 2023Harms from Increasingly Agentic Algorithmic Systemscited by 134
  9. 2022Goal Misgeneralization: Why Correct Specifications Aren't Enough For Correct Goalscited by 120
  10. 2019Optimal Policies Tend to Seek Powercited by 115
  11. 2022Is Power-Seeking AI an Existential Risk?cited by 101
  12. 2016The Off-Switch Gamecited by 97
  13. 2022The Fallacy of AI Functionalitycited by 88
  14. 2022Inverse scaling can become U-shapedcited by 76
  15. 2022Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governancecited by 67
  16. 2022Adversarial Training for High-Stakes Reliabilitycited by 60
  17. 2021Agent Incentives: A Causal Perspectivecited by 51
  18. 2022Structured access: an emerging paradigm for safe AI deploymentcited by 40
  19. 2022Discovering Agentscited by 28
  20. 2022What Do NLP Researchers Believe? Results of the NLP Community Metasurveycited by 28
  21. 2022From plane crashes to algorithmic harm: applicability of safety engineering frameworks for responsible MLcited by 26
  22. 2023Power-seeking can be probable and predictive for trained agentscited by 23
  23. 2022A Hazard Analysis Framework for Code Synthesis Large Language Modelscited by 22
  24. 2021Why and How Governments Should Monitor AI Developmentcited by 13
  25. 2023Exploring the Relevance of Data Privacy-Enhancing Technologies for AI Governance Use Casescited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 36; 2024: 53; 2025: 43; 2026: 35.

167 papers.

Sort:

By area:

  1. 2024OpenAI o1 System Cardcited by 1,220
  2. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
  3. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  4. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  5. 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 488
  6. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Promptscited by 368
  7. 2023Challenges and Applications of Large Language Modelscited by 251
  8. 2024Rethinking Machine Unlearning for Large Language Modelscited by 247
  9. 2023Large Language Model Alignment: A Surveycited by 241
  10. 2023AI Control: Improving Safety Despite Intentional Subversioncited by 207
  11. 2023Evaluating Large Language Models: A Comprehensive Surveycited by 200
  12. 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Setscited by 198
38 more
  1. 2023AI Deception: A Survey of Examples, Risks, and Potential Solutionscited by 185
  2. 2023Do-Not-Answer: A Dataset for Evaluating Safeguards in LLMscited by 176
  3. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraintscited by 174
  4. 2023Can LLM-Generated Misinformation Be Detected?cited by 173
  5. 2023Sociotechnical Safety Evaluation of Generative AI Systemscited by 154
  6. 2023Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchillacited by 145
  7. 2023Taken out of context: On measuring situational awareness in LLMscited by 131
  8. 2024AI Sandbagging: Language Models can Strategically Underperform on Evaluationscited by 131
  9. 2024Black-Box Access is Insufficient for Rigorous AI Auditscited by 123
  10. 2023Avalon's Game of Thoughts: Battle Against Deception through Recursive Contemplationcited by 102
  11. 2023A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directionscited by 101
  12. 2023Deception Abilities Emerged in Large Language Modelscited by 99
  13. 2023Frontier AI Regulation: Managing Emerging Risks to Public Safetycited by 95
  14. 2023Levels of AGI for Operationalizing Progress on the Path to AGIcited by 94
  15. 2023Evaluating Language-Model Agents on Realistic Autonomous Taskscited by 94
  16. 2024Red-Teaming for Generative AI: Silver Bullet or Security Theater?cited by 81
  17. 2024Visibility into AI Agentscited by 77
  18. 2023GPT4GEO: How a Language Model Sees the World's Geographycited by 68
  19. 2024Introducing v0.5 of the AI Safety Benchmark from MLCommonscited by 60
  20. 2024Machine Unlearning in Generative AI: A Surveycited by 49
  21. 2023Managing extreme AI risks amid rapid progresscited by 48
  22. 2024The Oscars of AI Theater: A Survey on Role-Playing with Language Modelscited by 47
  23. 2023Hoodwinked: Deception and Cooperation in a Text-Based Game for Language Modelscited by 46
  24. 2024Sabotage Evaluations for Frontier Modelscited by 45
  25. 2024Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)cited by 44
  26. 2023Welfare Diplomacy: Benchmarking Language Model Cooperationcited by 43
  27. 2023GPT-who: An Information Density-based Machine-Generated Text Detectorcited by 38
  28. 2024Extending Activation Steering to Broad Skills and Multiple Behaviourscited by 36
  29. 2025Model Tampering Attacks Enable More Rigorous Evaluations of LLM Capabilitiescited by 36
  30. 2024SafeSora: Towards Safety Alignment of Text2Video Generation via a Human Preference Datasetcited by 34
  31. 2023Open-Sourcing Highly Capable Foundation Models: An evaluation of risks, benefits, and alternative methods for pursuing open-source objectivescited by 31
  32. 2024Adversaries Can Misuse Combinations of Safe Modelscited by 31
  33. 2025Evaluating Frontier Models for Stealth and Situational Awarenesscited by 30
  34. 2024Farsight: Fostering Responsible AI Awareness During AI Application Prototypingcited by 28
  35. 2024Failures to Find Transferable Image Jailbreaks Between Vision-Language Modelscited by 27
  36. 2023Compositional Capabilities of Autoregressive Transformers: A Study on Synthetic, Interpretable Taskscited by 21
  37. 2024Two Types of AI Existential Risk: Decisive and Accumulativecited by 21
  38. 2024BeHonest: Benchmarking Honesty in Large Language Modelscited by 20

50 most cited of 167 All 167 (JSON)