mlsanityData to 14 Oct 2026

Do as I Say, Not as I Do: Instruction-Induction Conflict in LLMs

arXiv 2605.20382 ↗First posted 19 May 2026cs.CL cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Language models are trained to follow instructions, but they are also powerful pattern completers. What happens when these two objectives conflict? We construct conversations in which a user instruction to behave in a target way T (e.g., always output a specific token, answer in a particular language, or adopt a persona) is opposed by N hardcoded assistant turns demonstrating a competing pattern P. We then measure instruction-following (IF) rates in this setting, across 13 models and 16 different instructions, for up to 50 turns. Average instruction-following rates range from 1% to 99% across models, largely uncorrelated with standard capability benchmarks. The transition from instruction-following to pattern-following is universal but highly model-dependent. Robustness is modulated both by instruction content, with models resisting induction longer when instructions align with their trained value priors, and by output format, with diverse multi-token responses proving substantially more resistant than single-token outputs. Chain-of-thought reasoning improves robustness but does not eliminate susceptibility, and can produce dissociation between correct deliberation and incorrect output. When asked to predict their behavior in this setting, models achieve 83.5% accuracy on average but systematically underestimate their own resistance to induction pressure. These results suggest that instruction-following remains brittle under induction pressure even for otherwise capable models, and that output diversity, rather than semantic engagement with the input, is the primary factor predicting robustness.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 31 of its 36 references that are in the map. First posted on arXiv 19 May 2026.

Builds on

31 of 36 references.

References · 31

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  2. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  3. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  4. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  5. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  6. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  7. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  8. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  9. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 841
  10. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 832
  11. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 493
  12. 2023Larger language models do in-context learning differentlycited by 394
19 more
  1. 2024Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attackcited by 390
  2. 2023Function Vectors in Large Language Modelscited by 308
  3. 2025Generalizing Verifiable Instruction Followingcited by 194
  4. 2024Benchmarking Complex Instruction-Following with Multiple Constraints Compositioncited by 152
  5. 2024InFoBench: Evaluating Instruction Following Ability in Large Language Modelscited by 135
  6. 2024Looking Inward: Language Models Can Learn About Themselves by Introspectioncited by 113
  7. 2025Tell me about yourself: LLMs are aware of their learned behaviorscited by 91
  8. 2026The Assistant Axis: Situating and Stabilizing the Default Persona of Language Modelscited by 90
  9. 2024Me, Myself, and AI: The Situational Awareness Dataset (SAD) for LLMscited by 84
  10. 2026Emergent Introspective Awareness in Large Language Modelscited by 82
  11. 2023Overthinking the Truth: Understanding how Language Models Process False Demonstrationscited by 81
  12. 2023Adversarial Demonstration Attacks on Large Language Modelscited by 63
  13. 2025Which Attention Heads Matter for In-Context Learning?cited by 61
  14. 2023Can LLMs Follow Simple Rules?cited by 46
  15. 2023Hijacking Large Language Models via Adversarial In-Context Learningcited by 45
  16. 2025Self-Interpretability: LLMs Can Describe Complex Internal Processes that Drive Their Decisionscited by 25
  17. 2025Emergent Misalignment via In-Context Learning: Narrow in-context examples can produce broadly misaligned LLMscited by 17
  18. 2025Interpreting the Repeated Token Phenomenon in Large Language Modelscited by 16
  19. 2025Next-token pretraining implies in-context learningcited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Do LLMs "know" internally when they follow instructions?
  2. 2026Hierarchical Alignment: Enforcing Hierarchical Instruction-Following in LLMs through Logical Consistency
  3. 2025Can Language Models Follow Multiple Turns of Entangled Instructions?
  4. 2025ConInstruct: Evaluating Large Language Models on Conflict Detection and Resolution in Instructions
  5. 2025The Atomic Instruction Gap: Instruction-Tuned LLMs Struggle with Simple, Self-Contained Directives
  6. 2025The Instruction Gap: LLMs get lost in Following Instruction
  7. 2023Evaluating the Instruction-Following Robustness of Large Language Models to Prompt Injection
  8. 2023FollowEval: A Multi-Dimensional Benchmark for Assessing the Instruction-Following Capability of Large Language Models