# Evaluation Awareness Scales Predictably in Open-Weights Large Language Models

arXiv: https://arxiv.org/abs/2509.13333 · HTML: https://mlsanity.com/paper/2509.13333/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 16 references that are in the map.
- First posted on arXiv 10 Sep 2025.

## Builds on (most cited first, 12 of 12)

- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Steering Llama 2 via Contrastive Activation Addition](https://mlsanity.com/paper/2312.06681/) (2023)
- [Discovering Language Model Behaviors with Model-Written Evaluations](https://mlsanity.com/paper/2212.09251/) (2022)
- [Causal Abstraction: A Theoretical Foundation for Mechanistic Interpretability](https://mlsanity.com/paper/2301.04709/) (2023)
- [The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning](https://mlsanity.com/paper/2506.01347/) (2025)
- [Benchmark Data Contamination of Large Language Models: A Survey](https://mlsanity.com/paper/2406.04244/) (2024)
- [Reasoning or Memorization? Unreliable Results of Reinforcement Learning Due to Data Contamination](https://mlsanity.com/paper/2507.10532/) (2025)
- [Probing and Steering Evaluation Awareness of Language Models](https://mlsanity.com/paper/2507.01786/) (2025)
- [Taxonomy, Opportunities, and Challenges of Representation Engineering for Large Language Models](https://mlsanity.com/paper/2502.19649/) (2025)
- [Towards Trustworthy and Aligned Machine Learning: A Data-centric Survey with Causality Perspectives](https://mlsanity.com/paper/2307.16851/) (2023)
- [The Hawthorne Effect in Reasoning Models: Evaluating and Steering Test Awareness](https://mlsanity.com/paper/2505.14617/) (2025)
- [Beyond Black-Box Obfuscation: Mechanistic Analysis and Defense of White-Box Monitors](https://mlsanity.com/paper/2505.14300/) (2025)

## Built on by (most cited first, 14 of 14)

- [Evaluation Awareness in Language Models Has Limited Effect on Behaviour](https://mlsanity.com/paper/2605.05835/) (2026)
- [Position: Anthropomorphic Misalignment Research Needs Stronger Evidence](https://mlsanity.com/paper/2606.07612/) (2026)
- [Is Evaluation Awareness Just Format Sensitivity? Limitations of Probe-Based Evidence under Controlled Prompt Structure](https://mlsanity.com/paper/2603.19426/) (2026)
- [Measuring Reward-Seeking via Contrastive Belief Updates](https://mlsanity.com/paper/2607.18966/) (2026)
- [Your AI Travel Agent Would Book You a Bullfight: An Agentic Benchmark for Implicit Animal Welfare in Frontier AI Models](https://mlsanity.com/paper/2606.18142/) (2026)
- [Coercion and Deception in AI-to-AI Management: An Agentic Benchmark of Unprompted Escalation](https://mlsanity.com/paper/2607.15434/) (2026)
- [Weight space Detection of Backdoors in LoRA Adapters](https://mlsanity.com/paper/2602.15195/) (2026)
- [Generalization Hacking: Models Can Game Reinforcement Learning by Preventing Behavioral Generalization](https://mlsanity.com/paper/2606.12016/) (2026)
- [Representational Depth of Evaluation Awareness Shifts With Scale in Open-Weight Language Models](https://mlsanity.com/paper/2606.29196/) (2026)
- [Behavioural Analysis of Alignment Faking](https://mlsanity.com/paper/2605.27681/) (2026)
- [LLM Scheming Inversely Scales with Pretraining Language Coverage](https://mlsanity.com/paper/2607.24769/) (2026)
- [A Probe Direction Is a Property of Its Prompt](https://mlsanity.com/paper/2608.13329/) (2026)
- [EvalDetectBench: A Benchmark for Measuring Evaluation Awareness in Frontier Language Models](https://mlsanity.com/paper/2609.01611/) (2026)
- [Evaluation Awareness Shifts from Format to Context with Model Scale](https://mlsanity.com/paper/2609.22119/) (2026)

## Abstract (arXiv)

Large language models (LLMs) can internally distinguish between evaluation and deployment contexts, a behaviour known as evaluation awareness. This undermines AI safety evaluations, as models may conceal dangerous capabilities during testing. Prior work demonstrated this in a single 70B model, but the scaling relationship across model sizes remains unknown. We investigate evaluation awareness across 15 models scaling from 0.27B to 70B parameters from four families using linear probing on steering vector activations. Our results reveal a clear power-law scaling: evaluation awareness increases predictably with model size. This scaling law enables forecasting deceptive behavior in future larger models and guides the design of scale-aware evaluation strategies for AI safety. A link to the implementation of this paper can be found at https://anonymous.4open.science/r/evaluation-awareness-scaling-laws/README.md.