# The Hawthorne Effect in Reasoning Models: Evaluating and Steering Test Awareness

arXiv: https://arxiv.org/abs/2505.14617 · HTML: https://mlsanity.com/paper/2505.14617/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning.
- Builds on 18 of its 27 references that are in the map.
- First posted on arXiv 20 May 2025.

## Placed in

- LLM reasoning / [Test-time compute and efficient thinking](https://mlsanity.com/lr/test-time-compute-and-efficient-thinking/)
- LLM safety and security / [Safety alignment, refusal and guardrails](https://mlsanity.com/safety/safety-alignment-refusal-and-guardrails/)

## Builds on (most cited first, 18 of 18)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge](https://mlsanity.com/paper/1811.00937/) (2018)
- [HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal](https://mlsanity.com/paper/2402.04249/) (2024)
- [Representation Engineering: A Top-Down Approach to AI Transparency](https://mlsanity.com/paper/2310.01405/) (2023)
- [Steering Llama 2 via Contrastive Activation Addition](https://mlsanity.com/paper/2312.06681/) (2023)
- [Refusal in Language Models Is Mediated by a Single Direction](https://mlsanity.com/paper/2406.11717/) (2024)
- [BBQ: A Hand-Built Bias Benchmark for Question Answering](https://mlsanity.com/paper/2110.08193/) (2021)
- [AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents](https://mlsanity.com/paper/2410.09024/) (2024)
- [Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation](https://mlsanity.com/paper/2503.11926/) (2025)
- [Alignment faking in large language models](https://mlsanity.com/paper/2412.14093/) (2024)
- [Analyzing the Generalization and Reliability of Steering Vectors](https://mlsanity.com/paper/2407.12404/) (2024)
- [Personalized Steering of Large Language Models: Versatile Steering Vectors Through Bi-directional Preference Optimization](https://mlsanity.com/paper/2406.00045/) (2024)
- [Sabotage Evaluations for Frontier Models](https://mlsanity.com/paper/2410.21514/) (2024)
- [Taxonomy, Opportunities, and Challenges of Representation Engineering for Large Language Models](https://mlsanity.com/paper/2502.19649/) (2025)
- [Model Surgery: Modulating LLM's Behavior Via Simple Parameter Editing](https://mlsanity.com/paper/2407.08770/) (2024)

## Built on by (most cited first, 30 of 30)

- [Probing and Steering Evaluation Awareness of Language Models](https://mlsanity.com/paper/2507.01786/) (2025)
- [Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challenges](https://mlsanity.com/paper/2604.13602/) (2026)
- [Can We Predict Alignment Before Models Finish Thinking? Towards Monitoring Misaligned Reasoning Models](https://mlsanity.com/paper/2507.12428/) (2025)
- [Evaluation Awareness Scales Predictably in Open-Weights Large Language Models](https://mlsanity.com/paper/2509.13333/) (2025)
- [Neural Chameleons: Language Models Can Learn to Hide Their Thoughts from Unseen Activation Monitors](https://mlsanity.com/paper/2512.11949/) (2025)
- [Evaluation Awareness in Language Models Has Limited Effect on Behaviour](https://mlsanity.com/paper/2605.05835/) (2026)
- [Stateless Yet Not Forgetful: Implicit Memory as a Hidden Channel in LLMs](https://mlsanity.com/paper/2602.08563/) (2026)
- [UK AISI Alignment Evaluation Case-Study](https://mlsanity.com/paper/2604.00788/) (2026)
- [LURE: Live-Usage Replay Evaluations for Reducing Evaluation Awareness](https://mlsanity.com/paper/2605.26438/) (2026)
- [Models That Know How Evaluations Are Designed Score Safer](https://mlsanity.com/paper/2605.28591/) (2026)
- [Interaction Protocol Shapes Moral Judgment in Multi-Agent Debate](https://mlsanity.com/paper/2510.10002/) (2025)
- [CIAware-Bench: Benchmarking Control Intervention Awareness Across Frontier LLMs](https://mlsanity.com/paper/2606.11063/) (2026)
- [Evaluation Awareness Is Not One Capability: Evidence from Open Language Models](https://mlsanity.com/paper/2606.23583/) (2026)
- [Shall We Play a Game? Language Models for Open-ended Wargames](https://mlsanity.com/paper/2509.17192/) (2025)
- [Monitoring and Discovering Reward Hacking with Internal Representations during LLM Evaluations](https://mlsanity.com/paper/2609.19101/) (2026)
- [ATLAS: Constitution-Conditioned Latent Geometry and Redistribution Across Language Models and Neural Perturbation Data](https://mlsanity.com/paper/2604.17663/) (2026)
- [Realistic honeypot evaluations for scheming propensity](https://mlsanity.com/paper/2605.29729/) (2026)
- [Building Comparative Motivation Profiles with Instrumental Interventions](https://mlsanity.com/paper/2606.08243/) (2026)
- [Generalization Hacking: Models Can Game Reinforcement Learning by Preventing Behavioral Generalization](https://mlsanity.com/paper/2606.12016/) (2026)
- [Safety from Honesty in a Disinterested AI Predictor](https://mlsanity.com/paper/2606.29657/) (2026)
- [Alignment Risks from Capability-Seeking RL Training](https://mlsanity.com/paper/2602.12124/) (2026)
- [Disentangling Deception and Hallucination Failures in LLMs](https://mlsanity.com/paper/2602.14529/) (2026)
- [Naturalistic measure of social norms alignment](https://mlsanity.com/paper/2605.23420/) (2026)
- [Sycophancy Towards Researchers Drives Performative Misalignment](https://mlsanity.com/paper/2606.08629/) (2026)
- [Defeat Devices in AI Systems](https://mlsanity.com/paper/2606.28863/) (2026)
- [A Probe Direction Is a Property of Its Prompt](https://mlsanity.com/paper/2608.13329/) (2026)
- [INTENT-AS-A-TOOL Makes it Easy to Track Agentic Misalignment](https://mlsanity.com/paper/2608.27348/) (2026)
- [EvalDetectBench: A Benchmark for Measuring Evaluation Awareness in Frontier Language Models](https://mlsanity.com/paper/2609.01611/) (2026)
- [Norms at a Price: Why RL-Based Alignment Can Promise Conditional Compliance at Best](https://mlsanity.com/paper/2609.07627/) (2026)
- [Seeing Through the Forgery Is Not Enough: Regressive Sycophancy under Fabricated Evidence in Medical Vision-Language Models](https://mlsanity.com/paper/2610.17989/) (2026)

## Abstract (arXiv)

Reasoning-focused LLMs sometimes alter their behavior when they detect that they are being evaluated, which can lead them to optimize for test-passing performance or to comply more readily with harmful prompts if real-world consequences appear absent. We present the first quantitative study of how such "test awareness" impacts model behavior, particularly its performance on safety-related tasks. We introduce a white-box probing framework that (i) linearly identifies awareness-related activations and (ii) steers models toward or away from test awareness while monitoring downstream performance. We apply our method to different state-of-the-art open-weight reasoning LLMs across both realistic and hypothetical tasks (denoting tests or simulations). Our results demonstrate that test awareness significantly impacts safety alignment (such as compliance with harmful requests and conforming to stereotypes) with effects varying in both magnitude and direction across models. By providing control over this latent effect, our work aims to provide a stress-test mechanism and increase trust in how we perform safety evaluations.