# Linear Probe Accuracy Scales with Model Size and Benefits from Multi-Layer Ensembling

arXiv: https://arxiv.org/abs/2604.13386 · HTML: https://mlsanity.com/paper/2604.13386/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 13 of its 17 references that are in the map.
- First posted on arXiv 15 Apr 2026.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Safety alignment and jailbreak attacks](https://mlsanity.com/alignment/safety-alignment-and-jailbreak-attacks/)
- Mechanistic interpretability / [Probing deception, bias and beliefs](https://mlsanity.com/mechinterp/probing-deception-bias-and-beliefs/)

## Builds on (most cited first, 13 of 13)

- [Representation Engineering: A Top-Down Approach to AI Transparency](https://mlsanity.com/paper/2310.01405/) (2023)
- [AI Deception: A Survey of Examples, Risks, and Potential Solutions](https://mlsanity.com/paper/2308.14752/) (2023)
- [Agentic Misalignment: How LLMs Could Be Insider Threats](https://mlsanity.com/paper/2510.05179/) (2025)
- [Are Sparse Autoencoders Useful? A Case Study in Sparse Probing](https://mlsanity.com/paper/2502.16681/) (2025)
- [Universal Neurons in GPT2 Language Models](https://mlsanity.com/paper/2401.12181/) (2024)
- [Truth is Universal: Robust Detection of Lies in LLMs](https://mlsanity.com/paper/2407.12831/) (2024)
- [Detecting Strategic Deception Using Linear Probes](https://mlsanity.com/paper/2502.03407/) (2025)
- [The Neural Data Router: Adaptive Control Flow in Transformers Improves Systematic Generalization](https://mlsanity.com/paper/2110.07732/) (2021)
- [Building Production-Ready Probes For Gemini](https://mlsanity.com/paper/2601.11516/) (2026)
- [Constitutional Classifiers++: Efficient Production-Grade Defenses against Universal Jailbreaks](https://mlsanity.com/paper/2601.04603/) (2026)
- [Liars' Bench: Evaluating Lie Detectors for Language Models](https://mlsanity.com/paper/2511.16035/) (2025)
- [Difficulties with Evaluating a Deception Detector for AIs](https://mlsanity.com/paper/2511.22662/) (2025)
- [AI Deception: Risks, Dynamics, and Controls](https://mlsanity.com/paper/2511.22619/) (2025)

## Built on by (most cited first, 4 of 4)

- [Tool-Call Dependency Structure is Linearly Decodable in LLM Agent Residual Streams](https://mlsanity.com/paper/2605.25310/) (2026)
- [HalluTracer: Pre-Decoding Truthfulness Prediction via Depth-Averaged Probe-Logit](https://mlsanity.com/paper/2608.16353/) (2026)
- [Linear Probing Provides Robust and Efficient Detection of Machine-Generated Text](https://mlsanity.com/paper/2608.24780/) (2026)
- [Accuracy and Uncertainty in Machine Deception Detection (Research White Paper)](https://mlsanity.com/paper/2610.15314/) (2026)

## Abstract (arXiv)

Linear probes can detect when language models produce outputs they "know" are wrong, a capability relevant to both deception and reward hacking. However, single-layer probes are fragile: the best layer varies across models and tasks, and probes fail entirely on some deception types. We show that combining probes from multiple layers into an ensemble recovers strong performance even where single-layer probes fail, improving AUROC by +29% on Insider Trading and +78% on Harm-Pressure Knowledge. Across 12 models (0.5B--176B parameters), we find probe accuracy improves with scale: ~5% AUROC per 10x parameters (R=0.81). Geometrically, deception directions rotate gradually across layers rather than appearing at one location, explaining both why single-layer probes are brittle and why multi-layer ensembles succeed.