# Linear probes rely on textual evidence: Results from leakage mitigation studies in language models

arXiv: https://arxiv.org/abs/2509.21344 · HTML: https://mlsanity.com/paper/2509.21344/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Probing deception, bias and beliefs, one of 13 directions in Mechanistic interpretability.
- Builds on 20 of its 37 references that are in the map.
- First posted on arXiv 16 Sep 2025.

## Placed in

- Mechanistic interpretability / [Probing deception, bias and beliefs](https://mlsanity.com/mechinterp/probing-deception-bias-and-beliefs/)

## Builds on (most cited first, 20 of 20)

- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [Representation Engineering: A Top-Down Approach to AI Transparency](https://mlsanity.com/paper/2310.01405/) (2023)
- [Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations](https://mlsanity.com/paper/2312.06674/) (2023)
- [A General Language Assistant as a Laboratory for Alignment](https://mlsanity.com/paper/2112.00861/) (2021)
- [Training Large Language Models to Reason in a Continuous Latent Space](https://mlsanity.com/paper/2412.06769/) (2024)
- [LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Dataset](https://mlsanity.com/paper/2309.11998/) (2023)
- [Mechanistic Interpretability for AI Safety -- A Review](https://mlsanity.com/paper/2404.14082/) (2024)
- [Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation](https://mlsanity.com/paper/2503.11926/) (2025)
- [Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety](https://mlsanity.com/paper/2507.11473/) (2025)
- [How to Catch an AI Liar: Lie Detection in Black-Box LLMs by Asking Unrelated Questions](https://mlsanity.com/paper/2309.15840/) (2023)
- [Auditing language models for hidden objectives](https://mlsanity.com/paper/2503.10965/) (2025)
- [SHADE-Arena: Evaluating Sabotage and Monitoring in LLM Agents](https://mlsanity.com/paper/2506.15740/) (2025)
- [Detecting Strategic Deception Using Linear Probes](https://mlsanity.com/paper/2502.03407/) (2025)
- [International AI Safety Report](https://mlsanity.com/paper/2501.17805/) (2025)
- [Detecting High-Stakes Interactions with Activation Probes](https://mlsanity.com/paper/2506.10805/) (2025)
- [What Features in Prompts Jailbreak LLMs? Investigating the Mechanisms Behind Attacks](https://mlsanity.com/paper/2411.03343/) (2024)
- [Lessons from Studying Two-Hop Latent Reasoning](https://mlsanity.com/paper/2411.16353/) (2024)
- [Poser: Unmasking Alignment Faking LLMs by Manipulating Their Internals](https://mlsanity.com/paper/2405.05466/) (2024)
- [Combining Cost-Constrained Runtime Monitors for AI Safety](https://mlsanity.com/paper/2507.15886/) (2025)

## Built on by (most cited first, 2 of 2)

- [What Do Compliance Detectors Read? An Audit of Activation Probes and Guard Models](https://mlsanity.com/paper/2608.16852/) (2026)
- [Cross-Model In-Context Neurofeedback Reveals (Limited) Activation Control in LLMs](https://mlsanity.com/paper/2610.20179/) (2026)

## Abstract (arXiv)

White-box monitors are a popular technique for detecting potentially harmful behaviours in language models. While they perform well in general, their effectiveness in detecting text-ambiguous behaviour is disputed. In this work, we find evidence that removing textual evidence of a behaviour significantly decreases probe performance. The AUROC reduction ranges from 10- to 30-point depending on the setting. We evaluate probe monitors across three setups (Sandbagging, Sycophancy, and Bias), finding that when probes rely on textual evidence of the target behaviour (such as system prompts or CoT reasoning), performance degrades once these tokens are filtered. This filtering procedure is standard practice for output monitor evaluation. As further evidence of this phenomenon, we train Model Organisms which produce outputs without any behaviour verbalisations. We validate that probe performance on Model Organisms is substantially lower than unfiltered evaluations: 0.57 vs 0.74 AUROC for Bias, and 0.57 vs 0.94 AUROC for Sandbagging. Our findings suggest that linear probes may be brittle in scenarios where they must detect non-surface-level patterns.