mlsanityData to 14 Oct 2026

LLMs Can Covertly Sandbag on Capability Evaluations Against Chain-of-Thought Monitoring

arXiv 2508.00943 ↗First posted 31 Jul 2025cs.CR cs.AIcited by 18

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

Abstract (arXiv)

Trustworthy evaluations of dangerous capabilities are increasingly crucial for determining whether an AI system is safe to deploy. One empirically demonstrated threat is sandbagging - the strategic underperformance on evaluations by AI models or their developers. A promising defense is to monitor a model's chain-of-thought (CoT) reasoning, as this could reveal its intentions and plans. In this work, we measure the ability of models to sandbag on dangerous capability evaluations against a CoT monitor by prompting them to sandbag while being either monitor-oblivious or monitor-aware. We show that both frontier models and small open-sourced models can covertly sandbag against CoT monitoring 0-shot without hints. However, they cannot yet do so reliably: they bypass the monitor 16-36% of the time when monitor-aware, conditioned on sandbagging successfully. We qualitatively analyzed the uncaught CoTs to understand why the monitor failed. We reveal a rich attack surface for CoT monitoring and contribute five covert sandbagging policies generated by models. These results inform potential failure modes of CoT monitoring and may help build more diverse sandbagging model organisms.

Key facts. Cited by 18 papers in the map (data through 14 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 23 of its 28 references that are in the map. First posted on arXiv 31 Jul 2025.

Builds on

23 of 28 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety15 Jul
  2. 2025When Chain of Thought is Necessary, Language Models Struggle to Evade Monitors7 Jul
  3. 2025Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation14 Mar
  4. 2025Chain-of-Thought Reasoning In The Wild Is Not Always Faithful11 Mar
  5. 2023Measuring Faithfulness in Chain-of-Thought Reasoning17 Jul
  6. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting7 May

Other references · 17

  1. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  2. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  3. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  4. 2024OpenAI o1 System Cardcited by 1,244
  5. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 836
  6. 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 509
  7. 2024Deliberative Alignment: Reasoning Enables Safer Language Modelscited by 308
  8. 2024Frontier Models are Capable of In-context Schemingcited by 287
  9. 2023AI Control: Improving Safety Despite Intentional Subversioncited by 229
  10. 2024LAB-Bench: Measuring Capabilities of Language Models for Biology Researchcited by 156
  11. 2024AI Sandbagging: Language Models can Strategically Underperform on Evaluationscited by 145
  12. 2024Evaluating Frontier Models for Dangerous Capabilitiescited by 128
5 more
  1. 2025Are DeepSeek R1 And Other Reasoning Models More Faithful?cited by 72
  2. 2024Sabotage Evaluations for Frontier Modelscited by 52
  3. 2024Safety Cases: How to Justify the Safety of Advanced AI Systemscited by 48
  4. 2025Virology Capabilities Test (VCT): A Multimodal Virology Q&A Benchmarkcited by 31
  5. 2024The Probabilities Also Matter: A More Faithful Metric for Faithfulness of Free-Text Explanations in Large Language Modelscited by 29

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 14.

18 papers.

Sort:

By area:

  1. 2026Reasoning Models Struggle to Control their Chains of Thoughtcited by 25
  2. 2025Measuring Chain-of-Thought Monitorability Through Faithfulness and Verbositycited by 17
  3. 2025A Pragmatic Way to Measure Chain-of-Thought Monitorabilitycited by 13
  4. 2025All Code, No Thought: Current Language Models Struggle to Reason in Ciphered Languagecited by 8
  5. 2025State over Tokens: Characterizing the Role of Reasoning Tokenscited by 5
  6. 2026Coding with "Enemy": Can Human Developers Detect AI Agent Sabotage?cited by 4
  7. 2026Chain-of-Thought Monitoring Can Be Unreliable in Implicit-Influence Settingscited by 2
  8. 2026Bypassing AI Control Protocols via Agent-as-a-Proxy Attackscited by 1
  9. 2026Incentive-Aware AI Safety via Strategic Resource Allocation: A Stackelberg Security Games Perspectivecited by 1
  10. 2026Probing the Misaligned Thinking Process of Language Modelscited by 1
  11. 2026Option-Order Randomisation Reveals a Distributional Position Attractor in Prompted Sandbaggingcited by 0
  12. 2026Instruction Complexity Induces Positional Collapse in Adversarial LLM Evaluationcited by 0
6 more
  1. 2026CoT-Guard: Small Models for Strong Monitoringcited by 0
  2. 2026The Distributed Detectability Band Against Marginal-Preserving Attackscited by 0
  3. 2026The Arbiter Agent: Continually Monitoring Multi-Agent Conversations to Detect Emergent Misalignmentcited by 0
  4. 2026Defeat Devices in AI Systemscited by 0
  5. 2026Detecting Hidden Behaviors in LLMs via Activation-matched Finetuningcited by 0
  6. 2026On the Monitorability of Natural Language Autoencoders: Identification vs Discriminationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025CoT Red-Handed: Stress Testing Chain-of-Thought Monitoring
  2. 2025Monitoring Monitorability
  3. 2026Persuasion Attacks Can Decrease Effectiveness of CoT Monitoring
  4. 2026Training on Documents About Monitoring Leads to CoT Obfuscation
  5. 2026Evading Chain-of-Thought Monitoring Through Model Poisoning
  6. 2026Noticing the Watcher: LLM Agents Can Infer CoT Monitoring from Blocking Feedback
  7. 2025A Concrete Roadmap towards Safety Cases based on Chain-of-Thought Monitoring
  8. 2026Chain-of-thought obfuscation learned from output supervision can generalise to unseen tasks