# Prefill Awareness in Large Language Models

arXiv: https://arxiv.org/abs/2606.12747 · HTML: https://mlsanity.com/paper/2606.12747/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security.
- Builds on 23 of its 38 references that are in the map.
- First posted on arXiv 10 Jun 2026.

## Placed in

- LLM safety and security / [Safety alignment, refusal and guardrails](https://mlsanity.com/safety/safety-alignment-refusal-and-guardrails/)

## Builds on (most cited first, 23 of 23)

- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [Discovering Language Model Behaviors with Model-Written Evaluations](https://mlsanity.com/paper/2212.09251/) (2022)
- [Aligning AI With Shared Human Values](https://mlsanity.com/paper/2008.02275/) (2020)
- [OpenAssistant Conversations -- Democratizing Large Language Model Alignment](https://mlsanity.com/paper/2304.07327/) (2023)
- [Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks](https://mlsanity.com/paper/2404.02151/) (2024)
- [AI Control: Improving Safety Despite Intentional Subversion](https://mlsanity.com/paper/2312.06942/) (2023)
- [Taken out of context: On measuring situational awareness in LLMs](https://mlsanity.com/paper/2309.00667/) (2023)
- [Large Language Models Often Know When They Are Being Evaluated](https://mlsanity.com/paper/2505.23836/) (2025)
- [Emergent Introspective Awareness in Large Language Models](https://mlsanity.com/paper/2601.01828/) (2026)
- [Emotion Concepts and their Function in a Large Language Model](https://mlsanity.com/paper/2604.07729/) (2026)
- [Bypassing the Safety Training of Open-Source LLMs with Priming Attacks](https://mlsanity.com/paper/2312.12321/) (2023)
- [Constitutional Black-Box Monitoring for Scheming in LLM Agents](https://mlsanity.com/paper/2603.00829/) (2026)
- [Evaluating whether AI models would sabotage AI safety research](https://mlsanity.com/paper/2604.24618/) (2026)
- [Discovering Forbidden Topics in Language Models](https://mlsanity.com/paper/2505.17441/) (2025)
- [Steering Awareness: Detecting Activation Steering from Within](https://mlsanity.com/paper/2511.21399/) (2025)
- [Emergent Introspection in AI is Content-Agnostic](https://mlsanity.com/paper/2603.05414/) (2026)
- [UK AISI Alignment Evaluation Case-Study](https://mlsanity.com/paper/2604.00788/) (2026)
- [Endogenous Resistance to Activation Steering in Language Models](https://mlsanity.com/paper/2602.06941/) (2026)
- [Exposing the Systematic Vulnerability of Open-Weight Models to Prefill Attacks](https://mlsanity.com/paper/2602.14689/) (2026)
- [Self-Attribution Bias: When AI Monitors Go Easy on Themselves](https://mlsanity.com/paper/2603.04582/) (2026)
- [An Independent Safety Evaluation of Kimi K2.5](https://mlsanity.com/paper/2604.03121/) (2026)
- [Talent or Luck? Evaluating Attribution Bias in Large Language Models](https://mlsanity.com/paper/2505.22910/) (2025)

## Built on by (most cited first, 4 of 4)

- [LLMs Learn to Evade Latent Monitors from Prior Feedback Alone](https://mlsanity.com/paper/2609.36490/) (2026)
- [Prefilling the Reasoning Channel: Output-Prefix Attacks on Reasoning LLMs](https://mlsanity.com/paper/2609.29775/) (2026)
- [Training LLMs to Verbalize Evaluation Awareness](https://mlsanity.com/paper/2609.36316/) (2026)
- [Deception by Omission: Language Models Knowingly Hide Their Mistakes](https://mlsanity.com/paper/2610.11351/) (2026)

## Abstract (arXiv)

Safety-relevant studies of language models, including alignment and jailbreaking evaluations and AI control protocols, often rely on prefilling model outputs. If AI models can recognize and act on the fact their prior assistant messages have been inserted or edited, the effectiveness and validity of these methods could be compromised. We investigate whether frontier language models can distinguish between tampered and untampered assistant-side context, a capability we call prefill awareness. To do so, we construct a binary preference benchmark across three prefill mechanisms, filtering for cases where models show consistent stances. We find that frontier models show substantial prefill awareness: Claude Opus 4.5 detects prefills opposing its preferences in 9-35% of cases with a 0% false positive rate when prompted; additionally, models often revert towards baseline behavior without explicitly reporting that the prefill was foreign. Controlled ablations later also show that detection and resistance rely on different cues, where stylistic mismatch mainly affects whether models flag a prefill as foreign, while preference mismatch mainly affects whether they revert toward their baseline answer. We also examine more realistic agentic settings such as misalignment-continuation evaluations and SWE-bench trajectories, where frontier models sometimes disavow prefilled assistant turns in ways that depend strongly on dataset, task success, and hidden formatting artifacts. Our results indicate that prefill awareness is already a substantial confound for some prefill-based methods. We recommend that model developers track this capability in frontier systems.