Attention-Aligned Reasoning for Large Language Models
On the map: LLM reasoning → Reasoning benchmarks and evaluation
Abstract (arXiv)
Large Language Models (LLMs) tend to generate a long reasoning chain when solving complex tasks. However, as the reasoning chain extends, critical intermediate steps and the original prompt will be buried in the context, receiving insufficient attention and leading to errors. In this work, we present ATAR, a novel reasoning method that leverages the inherent reasoning structure to steer LLM attention. Our experiments show that ATAR outperforms SOTA methods across six benchmarks, achieving up to 15.39% absolute improvement. Furthermore, with ATAR, "non-reasoning" models achieve comparable or even better performance compared to reasoning models of the same size in most benchmarks. Finally, our ablation studies show that the attention alignment component contributes significantly, and that these improvements are persist under different attentionsteering backends.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 23 of its 70 references that are in the map. First posted on arXiv 3 Oct 2025.
Builds on
23 of 70 references.
Closest ancestors · 3
Same direction, newest first
- 2025Qwen3 Technical Report14 May
- 2025Phi-4-reasoning Technical Report30 Apr
- 2023LLaMA: Open and Efficient Foundation Language Models27 Feb
Other references · 20
- 2024The Llama 3 Herd of Modelscited by 14,715
- 2021Training Verifiers to Solve Math Word Problemscited by 10,109
- 2022Language Models (Mostly) Know What They Knowcited by 1,856
- 2024OpenAI o1 System Cardcited by 1,244
- 2024Phi-4 Technical Reportcited by 711
- 2025Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAscited by 521
- 2024Improve Mathematical Reasoning in Language Models by Automated Process Supervisioncited by 441
- 2024Massive Activations in Large Language Modelscited by 294
- 2025Deep Think with Confidencecited by 205
- 2025ReSum: Unlocking Long-Horizon Search Intelligence via Context Summarizationcited by 141
- 2023Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselvescited by 123
- 2022Compositional Semantic Parsing with Large Language Modelscited by 95
8 more
- 2023How FaR Are Large Language Models From Agents with Theory-of-Mind?cited by 76
- 2025Quantization Hurts Reasoning? An Empirical Study on Quantized Reasoning Modelscited by 73
- 2025RedStar: Does Scaling Long-CoT Data Unlock Better Slow-Reasoning Systems?cited by 60
- 2024Measuring and Controlling Instruction (In)Stability in Language Model Dialogscited by 53
- 2023TOAST: Transfer Learning via Attention Steeringcited by 15
- 2022Can Open-Domain QA Reader Utilize External Knowledge Efficiently like Humans?cited by 10
- 2025Quantization Meets Reasoning: Exploring and Mitigating Degradation of Low-Bit LLMs in Mathematical Reasoningcited by 7
- 2024Enhancing Trust in Large Language Models via Uncertainty-Calibrated Fine-Tuningcited by 5
Built on by
Citing papers per year: 2026: 2.
2 papers.
- 2026Steering Recurrent Reasoners at Inference Time with Readout Feedback25 Aug 2026
- 2026Code Generation by Differential Test Time Scaling19 May 2026
Similar, not cited
Close in wording, no citation either way
- 2024Attention Heads of Large Language Models: A Survey
- 2025Advancing Reasoning in Large Language Models: Promising Methods and Approaches
- 2025Attention Reveals More Than Tokens: Training-Free Long-Context Reasoning with Attention-guided Retrieval
- 2023Re-Reading Improves Reasoning in Large Language Models
- 2024Aggregation of Reasoning: A Hierarchical Framework for Enhancing Answer Selection in Large Language Models
- 2024A NotSo Simple Way to Beat Simple Bench
- 2024Extending Token Computation for LLM Reasoning
- 2026SPOT: Span-level Pause-of-Thought for Efficient and Interpretable Latent Reasoning in Large Language Models