mlsanityData to 7 Oct 2026

Measuring and Controlling Instruction (In)Stability in Language Model Dialogs

arXiv 2402.10962 ↗First posted 13 Feb 2024cs.CL cs.AI cs.LGcited by 51

Not on the map. Why?

Abstract (arXiv)

System-prompting is a standard tool for customizing language-model chatbots, enabling them to follow a specific instruction. An implicit assumption in the use of system prompts is that they will be stable, so the chatbot will continue to generate text according to the stipulated instructions for the duration of a conversation. We propose a quantitative benchmark to test this assumption, evaluating instruction stability via self-chats between two instructed chatbots. Testing popular models like LLaMA2-chat-70B and GPT-3.5, we reveal a significant instruction drift within eight rounds of conversations. An empirical and theoretical analysis of this phenomenon suggests the transformer attention mechanism plays a role, due to attention decay over long exchanges. To combat attention decay and instruction drift, we propose a lightweight method called split-softmax, which compares favorably against two strong baselines.

Key facts. Cited by 51 papers in the map (data through 7 Oct 2026). Builds on 30 of its 37 references that are in the map. First posted on arXiv 13 Feb 2024.

Builds on

30 of 37 references.

References · 30

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  6. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  7. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  8. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  9. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  10. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  11. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
  12. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
18 more
  1. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  2. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 793
  3. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 770
  4. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  5. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 553
  6. 2020GeDi: Generative Discriminator Guided Sequence Generationcited by 377
  7. 2023Function Vectors in Large Language Modelscited by 292
  8. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 290
  9. 2017Style Transfer from Non-Parallel Text by Cross-Alignmentcited by 246
  10. 2023Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Taskscited by 243
  11. 2021A Geometric Analysis of Neural Collapse with Unconstrained Featurescited by 226
  12. 2023Few-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluationcited by 134
  13. 2023Stay on topic with Classifier-Free Guidancecited by 90
  14. 2023System 2 Attention (is something you might need too)cited by 85
  15. 2023Can ChatGPT Write a Good Boolean Query for Systematic Review Literature Search?cited by 54
  16. 2023Instruction-following Evaluation through Verbalizer Manipulationcited by 30
  17. 2023Decision-Oriented Dialogue for Human-AI Collaborationcited by 29
  18. 2023Towards Better Evaluation of Instruction-Following: A Case-Study in Summarizationcited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 21; 2026: 28.

51 papers.

Sort:

By area:

  1. 2025Collab-Overcooked: Benchmarking and Evaluating Large Language Models as Collaborative Agentscited by 39
  2. 2024CopyBench: Measuring Literal and Non-Literal Reproduction of Copyright-Protected Text in Language Model Generationcited by 35
  3. 2025A Closer Look at System Prompt Robustnesscited by 31
  4. 2025Beyond Prompt Engineering: Robust Behavior Control in LLMs via Steering Target Atomscited by 23
  5. 2025Drift No More? Context Equilibria in Multi-Turn LLM Interactionscited by 21
  6. 2025Les Dissonances: Cross-Tool Harvesting and Polluting in Pool-of-Tools Empowered LLM Agentscited by 19
  7. 2025Position is Power: System Prompts as a Mechanism of Bias in Large Language Models (LLMs)cited by 19
  8. 2025Goal Alignment in LLM-Based User Simulators for Conversational AIcited by 19
  9. 2025Spotlight Your Instructions: Instruction-following with Dynamic Attention Steeringcited by 18
  10. 2025SOTOPIA-Omega: Dynamic Strategy Injection Learning and Social Instruction Following Evaluation for Social Agentscited by 16
  11. 2025Beyond the Surface: Enhancing LLM-as-a-Judge Alignment with Human via Internal Representationscited by 14
  12. 2025Persistent Personas? Role-Playing, Instruction Following, and Safety in Extended Interactionscited by 12
38 more
  1. 2025Focus Directions Make Your Language Models Pay More Attention to Relevant Contextscited by 9
  2. 2026SPASM: Stable Persona-driven Agent Simulation for Multi-turn Dialogue Generationcited by 9
  3. 2024An Auditing Test To Detect Behavioral Shift in Language Modelscited by 8
  4. 2026Attractor States Emerge in Multi-Turn LLM Conversationscited by 6
  5. 2025Reasoning Up the Instruction Ladder for Controllable Language Modelscited by 5
  6. 2026ContextEcho: A Benchmark for Persona Drift in Long Agentic-Coding Sessionscited by 5
  7. 2025When Harry Meets Superman: The Role of The Interlocutor in Persona-Based Dialogue Generationcited by 4
  8. 2025Psychometric Personality Shaping Modulates Capabilities and Safety in Language Modelscited by 4
  9. 2025When Instructions Multiply: Measuring and Estimating LLM Capabilities of Multiple Instructions Followingcited by 4
  10. 2026Mechanisms of Prompt-Induced Hallucination in Vision-Language Modelscited by 4
  11. 2026Personalization Increases Affective Alignment but Has Role-Dependent Effects on Epistemic Independence in LLMscited by 4
  12. 2026When Correct Beliefs Collapse: Epistemic Resilience of LLMs under Clinical Pressurecited by 4
  13. 2025A Comprehensive Review on Harnessing Large Language Models to Overcome Recommender System Challengescited by 3
  14. 2025Are LLM Agents Behaviorally Coherent? Latent Profiles for Social Simulationcited by 3
  15. 2025Attention-Aligned Reasoning for Large Language Modelscited by 2
  16. 2026HIPO: Instruction Hierarchy via Constrained Reinforcement Learningcited by 2
  17. 2025IROTE: Human-like Traits Elicitation of Large Language Model via In-Context Self-Reflective Optimizationcited by 1
  18. 2026Quantitative Introspection in Language Models: Tracking Emotive States Across Conversationcited by 1
  19. 2026Mitigating Misalignment Contagion by Steering with Implicit Traitscited by 1
  20. 2026Enabling KV Caching of Shared Prefix for Diffusion Language Modelscited by 1
  21. 2026Role Steering of Language Models for Social Simulationscited by 1
  22. 2026Entropy in Conversational AI: Structured Unpredictability as Inferrable Interioritycited by 1
  23. 2025Chatsparent: An Interactive System for Detecting and Mitigating Cognitive Fatigue in LLMscited by 0
  24. 2026Suffix-Constrained Greedy Search Algorithms for Causal Language Modelscited by 0
  25. 2026The Granularity Axis: A Micro-to-Macro Latent Direction for Social Roles in Language Modelscited by 0
  26. 2026Context Memorization for Efficient Long Context Generationcited by 0
  27. 2026Measuring What Persists: Conditioning Mechanisms and a Geometric Framework for AI Agent Identitycited by 0
  28. 2026On the Limits of Steering Vectors for Preference-Aligned Generationcited by 0
  29. 2026When Counterbalancing Hides the Bias: Access-Conditioned Position Lock in Forced-Choice LLM Evaluationcited by 0
  30. 2026AI YOU Town: Make Friends and Money with Your Digital Twincited by 0
  31. 2026Language Shapes Instruction Hierarchy Compliance in Multilingual LLMscited by 0
  32. 2026Dead text or binding clause? Measuring and restoring constraint influence in black-box LLM dialoguescited by 0
  33. 2026Compliance, Capability, and Conflict: Benchmarking Multimodal LLMs under System Messagescited by 0
  34. 2026What Does Multi-Agent LLM Debate Actually Change? A Layered Analysis of Disagreement and Answer Qualitycited by 0
  35. 2026ANIMASK: What the Model Contributes to Role Play in Simulated Story Worldscited by 0
  36. 2026Coding Agents with an Obstacle-Aware Harness for Safe Robot Manipulationcited by 0
  37. 2026Pretrained Persona Mixture Models and Tandem Models for Human Simulationcited by 0
  38. 2026Constitutional adapters: Inference-time interventions for misalignment and misusecited by 0

50 most cited of 51 All 51 (JSON)