mlsanityData to 14 Oct 2026

Sabotage Evaluations for Frontier Models

arXiv 2410.21514 ↗First posted 28 Oct 2024cs.LG cs.AI cs.CYcited by 52

Not on the map. Why?

Abstract (arXiv)

Sufficiently capable models could subvert human oversight and decision-making in important contexts. For example, in the context of AI development, models could covertly sabotage efforts to evaluate their own dangerous capabilities, to monitor their behavior, or to make decisions about their deployment. We refer to this family of abilities as sabotage capabilities. We develop a set of related threat models and evaluations. These evaluations are designed to provide evidence that a given model, operating under a given set of mitigations, could not successfully sabotage a frontier model developer or other large organization's activities in any of these ways. We demonstrate these evaluations on Anthropic's Claude 3 Opus and Claude 3.5 Sonnet models. Our results suggest that for these models, minimal mitigations are currently sufficient to address sabotage risks, but that more realistic evaluations and stronger mitigations seem likely to be necessary soon as capabilities improve. We also survey related evaluations we tried and abandoned. Finally, we discuss the advantages of mitigation-aware capability evaluations, and of simulating large-scale deployments using small-scale statistics.

Key facts. Cited by 52 papers in the map (data through 14 Oct 2026). Builds on 20 of its 50 references that are in the map. First posted on arXiv 28 Oct 2024.

Builds on

20 of 50 references.

References · 20

  1. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  2. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,207
  3. 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Trainingcited by 529
  4. 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 509
  5. 2024Debating with More Persuasive LLMs Leads to More Truthful Answerscited by 266
  6. 2019Risks from Learned Optimization in Advanced Machine Learning Systemscited by 258
  7. 2023AI Control: Improving Safety Despite Intentional Subversioncited by 229
  8. 2023An Overview of Catastrophic AI Riskscited by 222
  9. 2023Model evaluation for extreme riskscited by 177
  10. 2023Large Language Models can Strategically Deceive their Users when Put Under Pressurecited by 146
  11. 2024AI Sandbagging: Language Models can Strategically Underperform on Evaluationscited by 145
  12. 2023Purple Llama CyberSecEval: A Secure Coding Benchmark for Language Modelscited by 138
8 more
  1. 2024Evaluating Frontier Models for Dangerous Capabilitiescited by 128
  2. 2023Scheming AIs: Will AIs fake alignment during training in order to get power?cited by 97
  3. 2024Me, Myself, and AI: The Situational Awareness Dataset (SAD) for LLMscited by 84
  4. 2024Safety Cases: How to Justify the Safety of Advanced AI Systemscited by 48
  5. 2024Stress-Testing Capability Elicitation With Password-Locked Modelscited by 46
  6. 2024Measuring and Benchmarking Large Language Models' Capabilities to Generate Persuasive Languagecited by 31
  7. 2023TASRA: a Taxonomy and Analysis of Societal-Scale Risks from AIcited by 26
  8. 2024Rel-A.I.: An Interaction-Centered Approach To Measuring Human-LM Reliancecited by 16

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 20; 2026: 28.

52 papers.

Sort:

By area:

  1. 2024Alignment faking in large language modelscited by 360
  2. 2025Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safetycited by 249
  3. 2024RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human expertscited by 175
  4. 2025Natural Emergent Misalignment from Reward Hacking in Production RLcited by 136
  5. 2025When Chain of Thought is Necessary, Language Models Struggle to Evade Monitorscited by 88
  6. 2025A sketch of an AI control safety casecited by 39
  7. 2025Evaluating Frontier Models for Stealth and Situational Awarenesscited by 37
  8. 2025Probing and Steering Evaluation Awareness of Language Modelscited by 35
  9. 2025Reliable Weak-to-Strong Monitoring of LLM Agentscited by 31
  10. 2025The Hawthorne Effect in Reasoning Models: Evaluating and Steering Test Awarenesscited by 30
  11. 2024Noise Injection Reveals Hidden Capabilities of Sandbagging Language Modelscited by 24
  12. 2025The Elicitation Game: Evaluating Capability Elicitation Techniquescited by 18
38 more
  1. 2025LLMs Can Covertly Sandbag on Capability Evaluations Against Chain-of-Thought Monitoringcited by 18
  2. 2025Evaluation Faking: Unveiling Observer Effects in Safety Evaluation of Frontier AI Systemscited by 14
  3. 2026Monitoring Emergent Reward Hacking During Generation via Internal Activationscited by 14
  4. 2025The Singapore Consensus on Global AI Safety Research Prioritiescited by 10
  5. 2025Combining Cost-Constrained Runtime Monitors for AI Safetycited by 10
  6. 2025Do Large Language Models Know What They Are Capable Of?cited by 10
  7. 2025The Impact of Off-Policy Training Data on Probe Generalisationcited by 9
  8. 2026Measuring AI R&D Automationcited by 9
  9. 2026The Truthfulness Spectrum Hypothesiscited by 8
  10. 2026LinuxArena: A Control Setting for AI Agents in Live Production Software Environmentscited by 8
  11. 2026Evaluation Awareness in Language Models Has Limited Effect on Behaviourcited by 7
  12. 2025Advanced Financial Reasoning at Scale: A Comprehensive Evaluation of Large Language Models on CFA Level IIIcited by 6
  13. 2026Training Agents to Self-Report Misbehaviorcited by 6
  14. 2025OpenDeception: Learning Deception and Trust in Human-AI Interaction via Multi-Agent Simulationcited by 5
  15. 2025Manipulation Attacks by Misaligned AI: Risk Analysis and Safety Case Frameworkcited by 5
  16. 2026Distributed Attacks in Persistent-State AI Controlcited by 5
  17. 2026Removing Sandbagging in LLMs by Training with Weak Supervisioncited by 4
  18. 2026Training Deliberative Monitors for Black-Box Scheming Detectioncited by 4
  19. 2026Gram: Assessing sabotage propensities via automated alignment auditingcited by 4
  20. 2026Coding with "Enemy": Can Human Developers Detect AI Agent Sabotage?cited by 4
  21. 2025Typed Chain-of-Thought: A Curry-Howard Framework for Verifying LLM Reasoningcited by 3
  22. 2026Intent Laundering: AI Safety Datasets Are Not What They Seemcited by 3
  23. 2026An Independent Safety Evaluation of Kimi K2.5cited by 3
  24. 2026Decomposing and Measuring Evaluation Awarenesscited by 3
  25. 2025Against racing to AGI: Cooperation, deterrence, and catastrophic riskscited by 2
  26. 2024Quantifying detection rates for dangerous capabilities: a theoretical model of dangerous capability evaluationscited by 1
  27. 2026Large Language Models Persuade Without Planning Theory of Mindcited by 1
  28. 2026Realistic honeypot evaluations for scheming propensitycited by 1
  29. 2026(Mis)generalization of Helpful-only Fine-tuningcited by 1
  30. 2026Attack Selection in Agentic AI Control Evaluations Meaningfully Decreases Safetycited by 1
  31. 2026Generalization Hacking: Models Can Game Reinforcement Learning by Preventing Behavioral Generalizationcited by 1
  32. 2026Questionnaire Responses Do not Capture the Safety of AI Agentscited by 0
  33. 2026Simulating the Evolution of Alignment and Values in Machine Intelligencecited by 0
  34. 2026Honeypot Protocolcited by 0
  35. 2026Diffuse AI Control on Fuzzy Taskscited by 0
  36. 2026The Distributed Detectability Band Against Marginal-Preserving Attackscited by 0
  37. 2026Unsupervised Features Mining via Activation Geometrycited by 0
  38. 2026AI Persuasion as a Threat to Human Controlcited by 0

50 most cited of 52 All 52 (JSON)