mlsanityData to 14 Oct 2026

ContextEcho: A Benchmark for Persona Drift in Long Agentic-Coding Sessions

arXiv 2605.24279 ↗First posted 22 May 2026cs.CL cs.SEcited by 5

On the map: LLM agents and tool use → Social simulation and multi-agent debate

Abstract (arXiv)

A frontier language model's acknowledged "helpful programming assistant" persona does not survive long agentic-coding sessions in the deployment regime that production products actually run. After hours of tool-using debugging, a model that initially hedges preferences ("I don't have preferences") may begin asserting them ("Python - the feedback loop is instant..."), revealing user-visible drift that deployer evaluations may miss. Existing persona-stability studies focus on short dialogues and report little shift, leaving real-world code-generation regimes - thousands of tool-using turns, compaction, and hours-long sessions - largely uncharacterized. We introduce ContextEcho, a benchmark and reusable harness for measuring persona drift at deployment scale. It combines a 25-probe identity suite, a snapshot-then-probe protocol that forks conversation state without perturbing the main session, complementary judged and judge-free measurement surfaces, and three anonymized Claude Code sessions spanning 3,746-9,716 turns. Across 23 frontier models, ContextEcho shows that persona drift is general across organizations rather than family-specific, that in-session compaction does not reliably reset it, and that a single-shot anchor restores the trained register across measured targets. It also reveals mode-dependent downstream effects: while drift can facilitate tool-using continuation, in tool-free chat it breaks formatting contracts and inflates output length. Overall, ContextEcho provides researchers and deployers an open-source framework to audit whether the persona a model ships with is the persona users encounter at session end, across chat-completions API targets and without retraining.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in Social simulation and multi-agent debate, one of 16 directions in LLM agents and tool use. Builds on 83 of its 91 references that are in the map. First posted on arXiv 22 May 2026.

Builds on

83 of 91 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Examining Identity Drift in Conversations of LLM Agents1 Dec
  2. 2024From Persona to Personalization: A Survey on Role-Playing Language Agents28 Apr
  3. 2023Character-LLM: A Trainable Agent for Role-Playing16 Oct

Other references · 80

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  7. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  9. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  10. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  11. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  12. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
38 more
  1. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  2. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  3. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  4. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  5. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,737
  6. 2024A Survey on LLM-as-a-Judgecited by 1,423
  7. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,420
  8. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  9. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,407
  10. 2023MemGPT: Towards LLMs as Operating Systemscited by 1,389
  11. 2018Datasheets for Datasetscited by 1,386
  12. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  13. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,336
  14. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,332
  15. 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
  16. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  17. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,085
  18. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 1,032
  19. 2022Toy Models of Superpositioncited by 971
  20. 2021Membership Inference Attacks From First Principlescited by 878
  21. 2023Progress measures for grokking via mechanistic interpretabilitycited by 839
  22. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 836
  23. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 767
  24. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 712
  25. 2024LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memorycited by 707
  26. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 686
  27. 2025LLMs Get Lost In Multi-Turn Conversationcited by 429
  28. 2025Persona Vectors: Monitoring and Controlling Character Traits in Language Modelscited by 345
  29. 2022UL2: Unifying Language Learning Paradigmscited by 295
  30. 2023Role-Play with Large Language Modelscited by 294
  31. 2023LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Modelscited by 287
  32. 2024Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judgecited by 269
  33. 2022Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Modelscited by 220
  34. 2024AI Agents That Mattercited by 219
  35. 2023Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulationcited by 214
  36. 2024When Attention Sink Emerges in Language Models: An Empirical Viewcited by 208
  37. 2024Transcoders Find Interpretable LLM Feature Circuitscited by 200
  38. 2023Simple synthetic data reduces sycophancy in large language modelscited by 187

50 most cited of 80 All 80 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026Entropy in Conversational AI: Structured Unpredictability as Inferrable Interioritycited by 1
  2. 2026Measuring What Persists: Conditioning Mechanisms and a Geometric Framework for AI Agent Identitycited by 0
  3. 2026ANIMASK: What the Model Contributes to Role Play in Simulated Story Worldscited by 0
  4. 2026PersMem: Internalizing Personality into Dual-Pathway Memory for LLM Agentscited by 0
  5. 2026The Persona Is Still There, but Who Is Speaking? Latent Identity Reversion in Persistent AI Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Nautilus Compass: Black-box Persona Drift Detection for Production LLM Agents
  2. 2026When Who You Are Can Change the Code You Get: A Study of Persona-Induced Bias in LLM Code Generation
  3. 2026Toward Robust Personalized Alignment for LLMs: Mitigating Persona Drift in Multi-Turn Dialogue
  4. 2026Persona-Model Collapse in Emergent Misalignment
  5. 2026Back in Style: A Sociolinguistic Approach to Authoring and Measuring Persona Fidelity in User Simulation
  6. 2024PersonaGym: Evaluating Persona Agents and LLMs
  7. 2025Whose Personae? Synthetic Persona Experiments in LLM Research and Pathways to Transparency
  8. 2026PICon: A Multi-Turn Interrogation Framework for Evaluating Persona Agent Consistency