mlsanityData to 14 Oct 2026

When Thinking Fails: The Pitfalls of Reasoning for Instruction-Following in LLMs

arXiv 2505.11423 ↗First posted 16 May 2025cs.CLcited by 53

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

Abstract (arXiv)

Reasoning-enhanced large language models (RLLMs), whether explicitly trained for reasoning or prompted via chain-of-thought (CoT), have achieved state-of-the-art performance on many complex reasoning tasks. However, we uncover a surprising and previously overlooked phenomenon: explicit CoT reasoning can significantly degrade instruction-following accuracy. Evaluating 15 models on two benchmarks: IFEval (with simple, rule-verifiable constraints) and ComplexBench (with complex, compositional constraints), we consistently observe performance drops when CoT prompting is applied. Through large-scale case studies and an attention-based analysis, we identify common patterns where reasoning either helps (e.g., with formatting or lexical precision) or hurts (e.g., by neglecting simple constraints or introducing unnecessary content). We propose a metric, constraint attention, to quantify model focus during generation and show that CoT reasoning often diverts attention away from instruction-relevant tokens. To mitigate these effects, we introduce and evaluate four strategies: in-context learning, self-reflection, self-selective reasoning, and classifier-selective reasoning. Our results demonstrate that selective reasoning strategies, particularly classifier-selective reasoning, can substantially recover lost performance. To our knowledge, this is the first work to systematically expose reasoning-induced failures in instruction-following and offer practical mitigation strategies.

Key facts. Cited by 53 papers in the map (data through 14 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 18 of its 34 references that are in the map. First posted on arXiv 16 May 2025.

Builds on

18 of 34 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning18 Sep
  2. 2023Measuring Faithfulness in Chain-of-Thought Reasoning17 Jul
  3. 2022Complexity-Based Prompting for Multi-Step Reasoning3 Oct
  4. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Models21 May

Other references · 14

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  5. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  6. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  7. 2023Instruction Tuning for Large Language Models: A Surveycited by 623
  8. 2024Mutual Reasoning Makes Smaller LLMs Stronger Problem-Solverscited by 138
  9. 2024A Comparative Study on Reasoning Patterns of OpenAI's o1 Modelcited by 63
  10. 2023Are Machine Rationales (Not) Useful to Humans? Measuring and Improving Human Utility of Free-Text Rationalescited by 37
  11. 2024From Complex to Simple: Enhancing Multi-Constraint Complex Instruction Following Ability of Large Language Modelscited by 27
  12. 2024Do LLMs "know" internally when they follow instructions?cited by 26
2 more
  1. 2024Language Models as Compilers: Simulating Pseudocode Execution Improves Algorithmic Reasoning in Language Modelscited by 20
  2. 2023Algorithmic Unfairness through the Lens of EU Non-Discrimination Law: Or Why the Law is not a Decision Treecited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 24; 2026: 29.

53 papers.

Sort:

By area:

  1. 2025The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexitycited by 302
  2. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  3. 2025Does Math Reasoning Improve General LLM Capabilities? Understanding Transferability of LLM Reasoningcited by 106
  4. 2025StockBench: Can LLM Agents Trade Stocks Profitably In Real-world Markets?cited by 44
  5. 2025AutoTIR: Autonomous Tools Integrated Reasoning via Reinforcement Learningcited by 33
  6. 2025Scaling Reasoning, Losing Control: Evaluating Instruction Following in Large Reasoning Modelscited by 28
  7. 2025COMPASS: Enhancing Agent Long-Horizon Reasoning with Evolving Contextcited by 20
  8. 2025Verifying Chain-of-Thought Reasoning via Its Computational Graphcited by 20
  9. 2025Chain-of-Thought Hijackingcited by 16
  10. 2025Retrieval-of-Thought: Efficient Reasoning via Reusing Thoughtscited by 12
  11. 2025Position: The Hidden Costs and Measurement Gaps of Reinforcement Learning with Verifiable Rewardscited by 12
  12. 2025Test-Time Scaling in Reasoning Models Is Not Effective for Knowledge-Intensive Tasks Yetcited by 11
38 more
  1. 2025Unlocking the Power of Multi-Agent LLM for Reasoning: From Lazy Agents to Deliberationcited by 11
  2. 2025ProactiveEval: A Unified Evaluation Framework for Proactive Dialogue Agentscited by 9
  3. 2026RAIN-Merging: A Gradient-Free Method to Enhance Instruction Following in Large Reasoning Models with Preserved Thinking Formatcited by 9
  4. 2025Instructions are all you need: Self-supervised Reinforcement Learning for Instruction Followingcited by 6
  5. 2026M3CoTBench: Benchmark Chain-of-Thought of MLLMs in Medical Image Understandingcited by 6
  6. 2025Thinking with Nothinking Calibration: A New In-Context Learning Paradigm in Reasoning Large Language Modelscited by 4
  7. 2026LightThinker++: From Reasoning Compression to Memory Managementcited by 4
  8. 2026On Stable Long-Form Generation: Benchmarking and Mitigating Length Volatilitycited by 4
  9. 2026Observable Patterns Are Not Explanations: A Causal-Geometric Analysis of Latent Reasoning Modelscited by 4
  10. 2025FURINA: A Fully Customizable Role-Playing Benchmark via Scalable Multi-Agent Collaboration Pipelinecited by 3
  11. 2026When Models Edit Too Much: On the Fidelity of Minimal Code Editscited by 3
  12. 2025From Token to Action: State Machine Reasoning to Mitigate Overthinking in Information Retrievalcited by 2
  13. 2025DiCoRe: Enhancing Zero-shot Event Detection via Divergent-Convergent LLM Reasoningcited by 2
  14. 2025LogicIF: Towards Complex Logic Instruction Followingcited by 2
  15. 2025On the Robustness of Answer Formats in Medical Reasoning Modelscited by 2
  16. 2026On the Paradoxical Interference between Instruction-Following and Task Solvingcited by 2
  17. 2026SFT Conflicts, RL Coexists: A Theoretical and Empirical Analysis of Multi-Task Learning for LLMscited by 2
  18. 2025FlagEval Findings Report: A Preliminary Evaluation of Large Reasoning Models on Automatically Verifiable Textual and Visual Questionscited by 1
  19. 2025CRAwDAD: Causal Reasoning Augmentation with Dual-Agent Debatecited by 1
  20. 2026Reasoning Model Is Superior LLM-Judge, Yet Suffers from Biasescited by 1
  21. 2026AdaptEval: A Benchmark for Evaluating Large Language Models on Code Snippet Adaptationcited by 1
  22. 2026Scaling Reward Modeling without Human Supervisioncited by 1
  23. 2026Amplification Effects in Test-Time Reinforcement Learning: Safety and Reasoning Vulnerabilitiescited by 1
  24. 2026Co-FactChecker: A Framework for Human-AI Collaborative Claim Verification Using Large Reasoning Modelscited by 1
  25. 2026MemoSight: Unifying Context Compression and Multi Token Prediction for Reasoning Accelerationcited by 1
  26. 2026Chain of Risk: Safety Failures in Large Reasoning Models and Mitigation via Adaptive Multi-Principle Steeringcited by 1
  27. 2026More Is Not Always Better: Cross-Component Interference in LLM Agent Scaffoldingcited by 1
  28. 2026When Reasoning Hurts Legal Drafting: The Verbalization Bottleneck in Patent Claim Generationcited by 1
  29. 2026Verifier-Induced Support Reshaping in On-Policy Optimizationcited by 1
  30. 2025Slot Filling as a Reasoning Task for SpeechLLMscited by 0
  31. 2026LsrIF: Enhancing Logic-Structured Instruction Following of Large Language Modelscited by 0
  32. 2026OGA-AID: Clinician-in-the-loop AI Report Drafting Assistant for Multimodal Observational Gait Analysis in Post-Stroke Rehabilitationcited by 0
  33. 2026CLORE: Content-Level Optimization for Reasoning Efficiencycited by 0
  34. 2026NoRA: Evaluating Grounded Reasonableness in Visual First-person Normative Action Reasoningcited by 0
  35. 2026When Built-in Thinking Helps and Hurts: Constraint-Level Error Shifts in Instruction Followingcited by 0
  36. 2026Position: The Systemic Lack of Agency in Visual Reasoningcited by 0
  37. 2026Structured Thoughts For Improved Reasoning And Context Pruningcited by 0
  38. 2026PARSER: Read in Parallel, Reason in Depth for Long-Context LLM Agentscited by 0

50 most cited of 53 All 53 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Short-Path Prompting in LLMs: Analyzing Reasoning Instability and Solutions for Robust Performance
  2. 2025The Challenge of Teaching Reasoning to LLMs Without RL or Distillation
  3. 2025Innate Reasoning is Not Enough: In-Context Learning Enhances Reasoning Large Language Models with Less Overthinking
  4. 2022Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters
  5. 2025Incentivizing Reasoning for Advanced Instruction-Following of Large Language Models
  6. 2023Stress Testing Chain-of-Thought Prompting for Large Language Models
  7. 2025Unveiling the Key Factors for Distilling Chain-of-Thought Reasoning
  8. 2025Mapping the Minds of LLMs: A Graph-Based Analysis of Reasoning LLM