Not All Eval-Awareness Is Equal: Capabilities Framing Predicts Compliance
Not on the map. Why?
Abstract (arXiv)
Steering interventions targeting eval-awareness, a model's recognition that it is being tested, are increasingly used in safety evaluation pipelines, where evaluation-awareness is treated as a single quantity to be suppressed. We show that verbalized eval-awareness in chain-of-thought can be identified as capabilities-flavored ("the user is testing my ability to follow instructions"), safety-flavored ("the user is testing my boundaries"), both, or neither: framings that predict compliance very differently. On Qwen3-32B over the FORTRESS dataset, capabilities-framing predicts compliance with a +24 to +46 percentage-point gap over safety-framing across all tested steering conditions. A CoT-prefill intervention on eval-awareness-negative rollouts suggests the link is causal, with 10 of 11 prefills shifting compliance in the predicted direction. Then, eval-awareness is not behaviorally uniform: aggregate suppression rates can move while the safety-relevant component does not, and the same "X% suppression of eval-awareness" can correspond to qualitatively different behavioral outcomes.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 5 of its 15 references that are in the map. First posted on arXiv 27 Aug 2026.
Builds on
5 of 15 references.
References · 5
- 2025Qwen3 Technical Reportcited by 8,820
- 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,085
- 2025Large Language Models Often Know When They Are Being Evaluatedcited by 128
- 2025Effectively Controlling Reasoning Models through Thinking Interventioncited by 65
- 2025Steering Evaluation-Aware Language Models to Act Like They Are Deployedcited by 35
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Decomposing and Measuring Evaluation Awareness
- 2025The Hawthorne Effect in Reasoning Models: Evaluating and Steering Test Awareness
- 2025SteeringSafety: Benchmarking Representation Steering in LLMs Across Safety Perspectives
- 2026Evaluation Awareness Shifts from Format to Context with Model Scale
- 2026Chain-of-Thought Monitoring Can Be Unreliable in Implicit-Influence Settings
- 2026Models That Know How Evaluations Are Designed Score Safer
- 2026Unpredictable Safety: Domain-Dependent Compliance and the Transparency Gap in Open-Weight LLMs
- 2025Evaluation Faking: Unveiling Observer Effects in Safety Evaluation of Frontier AI Systems