mlsanityData to 14 Oct 2026

Back Attention: Understanding and Enhancing Multi-Hop Reasoning in Large Language Models

arXiv 2502.10835 ↗First posted 15 Feb 2025cs.CLcited by 23

On the map: LLM reasoning → Knowledge- and tool-augmented reasoning

Abstract (arXiv)

We investigate how large language models perform latent multi-hop reasoning in prompts like "Wolfgang Amadeus Mozart's mother's spouse is". To analyze this process, we introduce logit flow, an interpretability method that traces how logits propagate across layers and positions toward the final prediction. Using logit flow, we identify four distinct stages in single-hop knowledge prediction: (A) entity subject enrichment, (B) entity attribute extraction, (C) relation subject enrichment, and (D) relation attribute extraction. Extending this analysis to multi-hop reasoning, we find that failures often stem from the relation attribute extraction stage, where conflicting logits reduce prediction accuracy. To address this, we propose back attention, a novel mechanism that enables lower layers to leverage higher-layer hidden states from different positions during attention computation. With back attention, a 1-layer transformer achieves the performance of a 2-layer transformer. Applied to four LLMs, back attention improves accuracy on five reasoning datasets, demonstrating its effectiveness in enhancing latent multi-hop reasoning ability.

Key facts. Cited by 23 papers in the map (data through 14 Oct 2026). Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning. Builds on 43 of its 63 references that are in the map. First posted on arXiv 15 Feb 2025.

Builds on

43 of 63 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Do Large Language Models Latently Perform Multi-Hop Reasoning?26 Feb
  2. 2024Investigating Multi-Hop Factual Shortcuts in Knowledge Editing of Large Language Models19 Feb

Other references · 41

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  6. 2023Let's Verify Step by Stepcited by 3,789
  7. 2023Mistral 7Bcited by 3,164
  8. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  9. 2017Decoupled Weight Decay Regularizationcited by 1,831
  10. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,420
  11. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,407
  12. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
29 more
  1. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,165
  2. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 1,017
  3. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 991
  4. 2022Toy Models of Superpositioncited by 971
  5. 2022In-context Learning and Induction Headscited by 964
  6. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 878
  7. 2023Reasoning with Language Model is Planning with World Modelcited by 861
  8. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 557
  9. 2024Scaling and evaluating sparse autoencoderscited by 553
  10. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 532
  11. 2024Improve Mathematical Reasoning in Language Models by Automated Process Supervisioncited by 441
  12. 2016Solving General Arithmetic Word Problemscited by 360
  13. 2023Towards Best Practices of Activation Patching in Language Models: Metrics and Methodscited by 359
  14. 2022Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoningcited by 334
  15. 2024Chain of Thought Empowers Transformers to Solve Inherently Serial Problemscited by 260
  16. 2024Chain-of-Thought Reasoning Without Promptingcited by 210
  17. 2024Mutual Reasoning Makes Smaller LLMs Stronger Problem-Solverscited by 138
  18. 2022Analyzing Transformers in Embedding Spacecited by 132
  19. 2023Automatic Prompt Augmentation and Selection with Chain-of-Thought from Labeled Datacited by 119
  20. 2024Arithmetic Without Algorithms: Language Models Solve Math With a Bag of Heuristicscited by 97
  21. 2024Model Editing at Scale leads to Gradual and Catastrophic Forgettingcited by 93
  22. 2023A Mechanistic Interpretation of Arithmetic Reasoning in Language Models using Causal Mediation Analysiscited by 89
  23. 2024Hopping Too Late: Exploring the Limitations of Large Language Models on Multi-Hop Queriescited by 78
  24. 2022Polysemanticity and Capacity in Neural Networkscited by 77
  25. 2023Successor Heads: Recurring, Interpretable Attention Heads In The Wildcited by 77
  26. 2024pyvene: A Library for Understanding and Improving PyTorch Models via Interventionscited by 54
  27. 2024Understanding and Patching Compositional Reasoning in LLMscited by 46
  28. 2024NNsight and NDIF: Democratizing Access to Open-Weight Foundation Model Internalscited by 45
  29. 2023VISIT: Visualizing and Interpreting the Semantic Information Flow of Transformerscited by 32

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 12; 2026: 11.

23 papers.

Sort:

By area:

  1. 2025Stop Overthinking: A Survey on Efficient Reasoning for Large Language Modelscited by 464
  2. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 384
  3. 2025A Survey on Latent Reasoningcited by 81
  4. 2026Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Modelscited by 27
  5. 2025Encode, Think, Decode: Scaling test-time reasoning with recursive latent thoughtscited by 24
  6. 2025AudioLens: A Closer Look at Auditory Attribute Perception of Large Audio-Language Modelscited by 18
  7. 2026Do Latent-CoT Models Think Step-by-Step? A Mechanistic Study on Sequential Reasoning Taskscited by 12
  8. 2025CaKE: Circuit-aware Editing Enables Generalizable Knowledge Learnerscited by 9
  9. 2025Internal Chain-of-Thought: Empirical Evidence for Layer-wise Subtask Scheduling in LLMscited by 9
  10. 2025How Do Language Models Compose Functions?cited by 8
  11. 2025Locate-then-Merge: Neuron-Level Parameter Fusion for Mitigating Catastrophic Forgetting in Multimodal LLMscited by 7
  12. 2026NEAT: Neuron-Based Early Exit for Large Reasoning Modelscited by 3
11 more
  1. 2026The Topological Trouble With Transformerscited by 3
  2. 2025Catch Your Breath: Adaptive Computation for Self-Paced Sequence Productioncited by 2
  3. 2026Tracing Relational Knowledge Recall in Large Language Modelscited by 2
  4. 2025Efficiently Learning Branching Networks for Multitask Algorithmic Reasoningcited by 1
  5. 2026Opening the Black Box: A Survey on the Mechanisms of Multi-Step Reasoning in Large Language Modelscited by 1
  6. 2026The Diminishing Returns of Early-Exit Decoding in Modern LLMscited by 1
  7. 2026Factual Retrieval in LLMs Is a Redundant, Distributed and Non-Contiguous Processcited by 1
  8. 2025Limit Analysis for Symbolic Multi-step Reasoning Tasks with Information Propagation Rules Based on Transformerscited by 0
  9. 2026A Comparative Analysis of LLM Memorization at Statistical and Internal Levels: Cross-Model Commonalities and Model-Specific Signaturescited by 0
  10. 2026Why Knowing Both Hops Is Not Enough: Understanding Two-Hop Generalization in Language Modelscited by 0
  11. 2026Transformers Stop Thinking Too Early, and a Tiny LoRA Fixes Itcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Towards Interpreting Language Models: A Case Study in Multi-Hop Reasoning
  2. 2026Layer-Order Inversion: Rethinking Latent Multi-Hop Reasoning in Large Language Models
  3. 2024Large Language Models Still Face Challenges in Multi-Hop Reasoning with External Knowledge
  4. 2023Memory Injections: Correcting Multi-Hop Reasoning Failures during Inference in Transformer-Based Language Models
  5. 2026Failure Modes in Multi-Hop QA: The Weakest Link Effect and the Recognition Bottleneck
  6. 2024Enhancing Multi-hop Reasoning through Knowledge Erasure in Large Language Model Editing
  7. 2024Do Large Language Models Perform Latent Multi-Hop Reasoning without Exploiting Shortcuts?
  8. 2024Seemingly Plausible Distractors in Multi-Hop Reasoning: Are Large Language Models Attentive Readers?