mlsanityData to 14 Oct 2026

LURE: Live-Usage Replay Evaluations for Reducing Evaluation Awareness

arXiv 2605.26438 ↗First posted 8 Apr 2026cs.CL cs.AIcited by 4

Not on the map. Why?

Abstract (arXiv)

Large language models can recognize when they are being evaluated (evaluation awareness) and behave differently because of that, which undermines the validity of safety and alignment benchmarks. We propose LURE (Live-Usage Replay Evaluations), a method for constructing deployment-like evaluations by replaying realistic agentic interaction trajectories and appending evaluation prompt at the end. We also introduce an automated pipeline for measuring evaluation realism, combining detection of verbalized evaluation awareness and judge-model estimates of the probability of logs being an evaluation, and validate it on a large dataset of deployment and evaluation transcripts. We find that LURE-based evaluations are substantially less distinguishable from deployment than widely used benchmarks and synthetic evaluation generators, and can approach the realism of real conversations with users. We instantiate LURE in scheming, AI safety sabotage, and sycophancy settings. Our results suggest that evaluation realism is a crucial property of alignment benchmarks and should be reported alongside benchmark results, especially when such results are used in safety cases.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 21 of its 30 references that are in the map. First posted on arXiv 8 Apr 2026.

Builds on

21 of 30 references.

References · 21

  1. 2023Towards Understanding Sycophancy in Language Modelscited by 1,040
  2. 2024Frontier Models are Capable of In-context Schemingcited by 287
  3. 2019Risks from Learned Optimization in Advanced Machine Learning Systemscited by 258
  4. 2025Agentic Misalignment: How LLMs Could Be Insider Threatscited by 158
  5. 2025Natural Emergent Misalignment from Reward Hacking in Production RLcited by 136
  6. 2025ELEPHANT: Measuring and understanding social sycophancy in LLMscited by 129
  7. 2025Large Language Models Often Know When They Are Being Evaluatedcited by 128
  8. 2025Measuring Sycophancy of Language Models in Multi-turn Dialoguescited by 104
  9. 2023Scheming AIs: Will AIs fake alignment during training in order to get power?cited by 97
  10. 2025Stress Testing Deliberative Alignment for Anti-Scheming Trainingcited by 86
  11. 2024From Yes-Men to Truth-Tellers: Addressing Sycophancy in Large Language Models with Pinpoint Tuningcited by 54
  12. 2025Evaluating Frontier Models for Stealth and Situational Awarenesscited by 37
9 more
  1. 2025Steering Evaluation-Aware Language Models to Act Like They Are Deployedcited by 35
  2. 2024Towards evaluations-based safety cases for AI schemingcited by 33
  3. 2025Evaluating the Paperclip Maximizer: Are RL-Based Language Models More Likely to Pursue Instrumental Goals?cited by 33
  4. 2025The Hawthorne Effect in Reasoning Models: Evaluating and Steering Test Awarenesscited by 30
  5. 2025DarkBench: Benchmarking Dark Patterns in Large Language Modelscited by 28
  6. 2025AgentMisalignment: Measuring the Propensity for Misaligned Behaviour in LLM-Based Agentscited by 22
  7. 2025PropensityBench: Evaluating Latent Safety Risks in Large Language Models via an Agentic Approachcited by 22
  8. 2024Safety cases for frontier AIcited by 18
  9. 2025Spilling the Beans: Teaching LLMs to Self-Report Their Hidden Objectivescited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Improving Evaluation Realism with Inference-Time Compute and Deployment Scaffoldscited by 0
  2. 2026TraceDance: An Automated System for Building Agent Behavior Benchmarks from Real-World Agent Deployment Tracescited by 0
  3. 2026Anatomy of LLM Sycophancy: What a Flip Rate Hidescited by 0
  4. 2026Seeing Through the Forgery Is Not Enough: Regressive Sycophancy under Fabricated Evidence in Medical Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Probing and Steering Evaluation Awareness of Language Models
  2. 2026AgentBoundary: Counterfactual Evaluation of Safety in Tool-Using LLM Agents
  3. 2025Probe-Rewrite-Evaluate: A Workflow for Reliable Benchmarks and Quantifying Evaluation Awareness
  4. 2026Claw-Eval: Towards Trustworthy Evaluation of Autonomous Agents
  5. 2025Evaluation Faking: Unveiling Observer Effects in Safety Evaluation of Frontier AI Systems
  6. 2026Reasoning Consistency Scanning: A Framework for Auditing Chain-of-Thought Validity in AI Safety Evaluations
  7. 2026Evaluating Agentic AI in the Wild: Failure Modes, Drift Patterns, and a Production Evaluation Framework
  8. 2026PASTABench: Proactive Assessment of Sequential Trajectories for Agent Safety