# Activation Probes Surface Code-Security Signals that the Model's Output Misses

arXiv: https://arxiv.org/abs/2608.09643 · HTML: https://mlsanity.com/paper/2608.09643/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 20 of its 26 references that are in the map.
- First posted on arXiv 10 Aug 2026.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 20 of 20)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Understanding intermediate layers using linear classifier probes](https://mlsanity.com/paper/1610.01644/) (2016)
- [Qwen2.5-Coder Technical Report](https://mlsanity.com/paper/2409.12186/) (2024)
- [DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence](https://mlsanity.com/paper/2401.14196/) (2024)
- [The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets](https://mlsanity.com/paper/2310.06824/) (2023)
- [The Internal State of an LLM Knows When It's Lying](https://mlsanity.com/paper/2304.13734/) (2023)
- [LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinations](https://mlsanity.com/paper/2410.02707/) (2024)
- [Large Language Models for Code: Security Hardening and Adversarial Testing](https://mlsanity.com/paper/2302.05319/) (2023)
- [Vulnerability Detection with Code Language Models: How Far Are We?](https://mlsanity.com/paper/2403.18624/) (2024)
- [The Adverse Effects of Code Duplication in Machine Learning Models of Code](https://mlsanity.com/paper/1812.06469/) (2018)
- [LLMs Cannot Reliably Identify and Reason About Security Vulnerabilities (Yet?): A Comprehensive Evaluation, Framework, and Benchmarks](https://mlsanity.com/paper/2312.12575/) (2023)
- [Detecting High-Stakes Interactions with Activation Probes](https://mlsanity.com/paper/2506.10805/) (2025)
- [NNsight and NDIF: Democratizing Access to Open-Weight Foundation Model Internals](https://mlsanity.com/paper/2407.14561/) (2024)
- [Security Weaknesses of Copilot-Generated Code in GitHub Projects: An Empirical Study](https://mlsanity.com/paper/2310.02059/) (2023)
- [PATCHEVAL: A New Benchmark for Evaluating LLMs on Patching Real-World Vulnerabilities](https://mlsanity.com/paper/2511.11019/) (2025)
- [CASTLE: Benchmarking Dataset for Static Code Analyzers and LLMs towards CWE Detection](https://mlsanity.com/paper/2503.09433/) (2025)
- [LPASS: Linear Probes as Stepping Stones for vulnerability detection using compressed LLMs](https://mlsanity.com/paper/2505.24451/) (2025)
- [AI Code in the Wild: Measuring Security Risks and Ecosystem Shifts of AI-Generated Code in Modern Software](https://mlsanity.com/paper/2512.18567/) (2025)

## Abstract (arXiv)

AI coding agents now write a growing share of production code, and human security review does not scale at the rate code is generated. The agents in widest use are closed-weight, so a deploying team cannot read their internals. It can instead run an open-weight model as a reviewer over the agent's output. That reviewer's activations are readable. We ask whether reading those activations recovers a security signal that simply asking the same reviewer misses. We fit a single linear probe per model on a corpus of paired vulnerable-and-fixed Python functions, then test it without retraining on real disclosed vulnerabilities whose weakness type the probe never saw in training, across five open-weight reviewer models. On the vulnerabilities fixed by changing a single function, the probe scores the vulnerable function above its fix on 61-67% of cases for every model, beating the 50% chance line. It also beats the same model's prompted YES/NO win-rate read from its logits, under every prompt we try. Asking the model for a written verdict, even with chain-of-thought, returns the same answer on the vulnerable and fixed function most of the time and so cannot tell them apart. Model activations carry a code-security signal that prompting the same model misses.