# LLMs Can Covertly Sandbag on Capability Evaluations Against Chain-of-Thought Monitoring

arXiv: https://arxiv.org/abs/2508.00943 · HTML: https://mlsanity.com/paper/2508.00943/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning.
- Builds on 23 of its 28 references that are in the map.
- First posted on arXiv 31 Jul 2025.

## Placed in

- LLM reasoning / [Prompting and chain-of-thought reasoning](https://mlsanity.com/lr/prompting-and-chain-of-thought-reasoning/)

## Builds on (most cited first, 23 of 23)

- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [OpenAI o1 System Card](https://mlsanity.com/paper/2412.16720/) (2024)
- [Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting](https://mlsanity.com/paper/2305.04388/) (2023)
- [Discovering Language Model Behaviors with Model-Written Evaluations](https://mlsanity.com/paper/2212.09251/) (2022)
- [Measuring Faithfulness in Chain-of-Thought Reasoning](https://mlsanity.com/paper/2307.13702/) (2023)
- [The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning](https://mlsanity.com/paper/2403.03218/) (2024)
- [Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation](https://mlsanity.com/paper/2503.11926/) (2025)
- [Deliberative Alignment: Reasoning Enables Safer Language Models](https://mlsanity.com/paper/2412.16339/) (2024)
- [Frontier Models are Capable of In-context Scheming](https://mlsanity.com/paper/2412.04984/) (2024)
- [Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety](https://mlsanity.com/paper/2507.11473/) (2025)
- [AI Control: Improving Safety Despite Intentional Subversion](https://mlsanity.com/paper/2312.06942/) (2023)
- [Chain-of-Thought Reasoning In The Wild Is Not Always Faithful](https://mlsanity.com/paper/2503.08679/) (2025)
- [LAB-Bench: Measuring Capabilities of Language Models for Biology Research](https://mlsanity.com/paper/2407.10362/) (2024)
- [AI Sandbagging: Language Models can Strategically Underperform on Evaluations](https://mlsanity.com/paper/2406.07358/) (2024)
- [Evaluating Frontier Models for Dangerous Capabilities](https://mlsanity.com/paper/2403.13793/) (2024)
- [When Chain of Thought is Necessary, Language Models Struggle to Evade Monitors](https://mlsanity.com/paper/2507.05246/) (2025)
- [Are DeepSeek R1 And Other Reasoning Models More Faithful?](https://mlsanity.com/paper/2501.08156/) (2025)
- [Sabotage Evaluations for Frontier Models](https://mlsanity.com/paper/2410.21514/) (2024)
- [Safety Cases: How to Justify the Safety of Advanced AI Systems](https://mlsanity.com/paper/2403.10462/) (2024)
- [Virology Capabilities Test (VCT): A Multimodal Virology Q&A Benchmark](https://mlsanity.com/paper/2504.16137/) (2025)
- [The Probabilities Also Matter: A More Faithful Metric for Faithfulness of Free-Text Explanations in Large Language Models](https://mlsanity.com/paper/2404.03189/) (2024)

## Built on by (most cited first, 18 of 18)

- [Reasoning Models Struggle to Control their Chains of Thought](https://mlsanity.com/paper/2603.05706/) (2026)
- [Measuring Chain-of-Thought Monitorability Through Faithfulness and Verbosity](https://mlsanity.com/paper/2510.27378/) (2025)
- [A Pragmatic Way to Measure Chain-of-Thought Monitorability](https://mlsanity.com/paper/2510.23966/) (2025)
- [All Code, No Thought: Current Language Models Struggle to Reason in Ciphered Language](https://mlsanity.com/paper/2510.09714/) (2025)
- [State over Tokens: Characterizing the Role of Reasoning Tokens](https://mlsanity.com/paper/2512.12777/) (2025)
- [Coding with "Enemy": Can Human Developers Detect AI Agent Sabotage?](https://mlsanity.com/paper/2606.05647/) (2026)
- [Chain-of-Thought Monitoring Can Be Unreliable in Implicit-Influence Settings](https://mlsanity.com/paper/2608.04735/) (2026)
- [Bypassing AI Control Protocols via Agent-as-a-Proxy Attacks](https://mlsanity.com/paper/2602.05066/) (2026)
- [Incentive-Aware AI Safety via Strategic Resource Allocation: A Stackelberg Security Games Perspective](https://mlsanity.com/paper/2602.07259/) (2026)
- [Probing the Misaligned Thinking Process of Language Models](https://mlsanity.com/paper/2606.24251/) (2026)
- [Option-Order Randomisation Reveals a Distributional Position Attractor in Prompted Sandbagging](https://mlsanity.com/paper/2604.26206/) (2026)
- [Instruction Complexity Induces Positional Collapse in Adversarial LLM Evaluation](https://mlsanity.com/paper/2604.27249/) (2026)
- [CoT-Guard: Small Models for Strong Monitoring](https://mlsanity.com/paper/2605.12746/) (2026)
- [The Distributed Detectability Band Against Marginal-Preserving Attacks](https://mlsanity.com/paper/2606.10456/) (2026)
- [The Arbiter Agent: Continually Monitoring Multi-Agent Conversations to Detect Emergent Misalignment](https://mlsanity.com/paper/2606.10747/) (2026)
- [Defeat Devices in AI Systems](https://mlsanity.com/paper/2606.28863/) (2026)
- [Detecting Hidden Behaviors in LLMs via Activation-matched Finetuning](https://mlsanity.com/paper/2609.00351/) (2026)
- [On the Monitorability of Natural Language Autoencoders: Identification vs Discrimination](https://mlsanity.com/paper/2610.18727/) (2026)

## Abstract (arXiv)

Trustworthy evaluations of dangerous capabilities are increasingly crucial for determining whether an AI system is safe to deploy. One empirically demonstrated threat is sandbagging - the strategic underperformance on evaluations by AI models or their developers. A promising defense is to monitor a model's chain-of-thought (CoT) reasoning, as this could reveal its intentions and plans. In this work, we measure the ability of models to sandbag on dangerous capability evaluations against a CoT monitor by prompting them to sandbag while being either monitor-oblivious or monitor-aware. We show that both frontier models and small open-sourced models can covertly sandbag against CoT monitoring 0-shot without hints. However, they cannot yet do so reliably: they bypass the monitor 16-36% of the time when monitor-aware, conditioned on sandbagging successfully. We qualitatively analyzed the uncaught CoTs to understand why the monitor failed. We reveal a rich attack surface for CoT monitoring and contribute five covert sandbagging policies generated by models. These results inform potential failure modes of CoT monitoring and may help build more diverse sandbagging model organisms.