mlsanityData to 14 Oct 2026

When Built-in Thinking Helps and Hurts: Constraint-Level Error Shifts in Instruction Following

arXiv 2606.09662 ↗First posted 8 Jun 2026cs.CLcited by 0

On the map: LLM reasoning → Mathematical reasoning

Abstract (arXiv)

Large reasoning models (LRMs) often improve math and coding performance, but their effect on instruction following is unclear. We study IFEval with Qwen3 models (1.7B-32B), using same-weights Thinking ON/OFF controls; four Hunyuan models provide directional cross-family support. Aggregate pass-rate changes are small (-0.55 to -3.52 pp), yet 10-20% of prompts switch between pass and fail across modes, suggesting that thinking changes the pattern of errors--some prompts improve while others worsen--rather than uniformly degrading performance. Under a post-hoc Qwen3-derived grouping, constraint types separate into Planning (global counting, structure, coordination), which improves at the class level under thinking, and Precision (exact local form), which consistently worsens; the class-level Planning/Precision sign pattern holds directionally for all four Hunyuan models despite Hunyuan's opposite aggregate direction. Thinking also changes final-answer length; matched-length analyses substantially reduce the Precision drop, but a residual penalty remains. Analyzing thinking traces with a cross-encoder relevance metric reveals three patterns: Neutral shows a positive relevance-compliance link (r approximately 0.15); Planning shows near-zero predictive correlation (r approximately 0.02) despite measurable trace engagement, consistent with an execution gap between CE-measured trace relevance and final-answer compliance; Precision shows a small negative correlation (r approximately -0.05), with failing instances having higher mean relevance than passing ones. Activation patching across four model sizes (1.7B-14B) shows that Precision flip instances are more often restored than Planning flip instances (32-58% vs. 14-40% mean layer-restoration), with the largest gap at 14B (about 30 pp).

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Mathematical reasoning, one of 17 directions in LLM reasoning. Builds on 11 of its 11 references that are in the map. First posted on arXiv 8 Jun 2026.

Builds on

11 of 11 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025Can LLMs Simulate Personas with Reversed Performance? A Systematic Investigation for Counterfactual Instruction Following in Math Reasoning Context8 Apr

Other references · 10

  1. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  2. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  3. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,605
  4. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 886
  5. 2024To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoningcited by 263
  6. 2024Benchmarking Complex Instruction-Following with Multiple Constraints Compositioncited by 152
  7. 2024InFoBench: Evaluating Instruction Following Ability in Large Language Modelscited by 135
  8. 2025When Thinking Fails: The Pitfalls of Reasoning for Instruction-Following in LLMscited by 53
  9. 2024Thinking LLMs: General Instruction Following with Thought Generationcited by 45
  10. 2025ReasonIF: Large Reasoning Models Fail to Follow Instructions During Reasoningcited by 13

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026To Think or Not To Think, That is The Question for Large Reasoning Models in Theory of Mind Tasks
  2. 2025Can Large Reasoning Models Improve Accuracy on Mathematical Tasks Using Flawed Thinking?
  3. 2025When Can Large Reasoning Models Save Thinking? Mechanistic Analysis of Behavioral Divergence in Reasoning
  4. 2025Scaling Reasoning, Losing Control: Evaluating Instruction Following in Large Reasoning Models
  5. 2025What Makes a Good Curriculum? Disentangling the Effects of Data Ordering on LLM Mathematical Reasoning
  6. 2025Reasoning Model is Stubborn: Diagnosing Instruction Overriding in Reasoning Models
  7. 2026Bridging Auxiliary Constraints to Resolve Instruction Following in Large Reasoning Models
  8. 2025ThinkPatterns-21k: A Systematic Study on the Impact of Thinking Patterns in LLMs