# Layer-Order Inversion: Rethinking Latent Multi-Hop Reasoning in Large Language Models

arXiv: https://arxiv.org/abs/2601.03542 · HTML: https://mlsanity.com/paper/2601.03542/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning.
- Builds on 39 of its 45 references that are in the map.
- First posted on arXiv 7 Jan 2026.

## Placed in

- LLM reasoning / [Knowledge- and tool-augmented reasoning](https://mlsanity.com/lr/knowledge-and-tool-augmented-reasoning/)

## Builds on (most cited first, 30 of 39)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [Transformer Feed-Forward Layers Are Key-Value Memories](https://mlsanity.com/paper/2012.14913/) (2020)
- [Measuring and Narrowing the Compositionality Gap in Language Models](https://mlsanity.com/paper/2210.03350/) (2022)
- [Mass-Editing Memory in a Transformer](https://mlsanity.com/paper/2210.07229/) (2022)
- [Probing Classifiers: Promises, Shortcomings, and Advances](https://mlsanity.com/paper/2102.12452/) (2021)
- [Eliciting Latent Predictions from Transformers with the Tuned Lens](https://mlsanity.com/paper/2303.08112/) (2023)
- [Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space](https://mlsanity.com/paper/2203.14680/) (2022)
- [Dissecting Recall of Factual Associations in Auto-Regressive Language Models](https://mlsanity.com/paper/2304.14767/) (2023)
- [MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop Questions](https://mlsanity.com/paper/2305.14795/) (2023)
- [Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Models](https://mlsanity.com/paper/2408.00724/) (2024)
- [AlphaEdit: Null-Space Constrained Knowledge Editing for Language Models](https://mlsanity.com/paper/2410.02355/) (2024)
- [Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models](https://mlsanity.com/paper/2401.06102/) (2024)
- [When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method](https://mlsanity.com/paper/2402.17193/) (2024)
- [PMET: Precise Model Editing in a Transformer](https://mlsanity.com/paper/2308.08742/) (2023)
- [Do Large Language Models Latently Perform Multi-Hop Reasoning?](https://mlsanity.com/paper/2402.16837/) (2024)
- [Evaluation of Retrieval-Augmented Generation: A Survey](https://mlsanity.com/paper/2405.07437/) (2024)
- [PlanGenLLMs: A Modern Survey of LLM Planning Capabilities](https://mlsanity.com/paper/2502.11221/) (2025)
- [Hopping Too Late: Exploring the Limitations of Large Language Models on Multi-Hop Queries](https://mlsanity.com/paper/2406.12775/) (2024)
- [The Benefits of a Concise Chain of Thought on Problem-Solving in Large Language Models](https://mlsanity.com/paper/2401.05618/) (2024)
- [Towards a Mechanistic Interpretation of Multi-Step Reasoning Capabilities of Language Models](https://mlsanity.com/paper/2310.14491/) (2023)
- [Understanding and Patching Compositional Reasoning in LLMs](https://mlsanity.com/paper/2402.14328/) (2024)
- [Knowledge Graph Enhanced Large Language Model Editing](https://mlsanity.com/paper/2402.13593/) (2024)
- [Locate-then-edit for Multi-hop Factual Recall under Knowledge Editing](https://mlsanity.com/paper/2410.06331/) (2024)
- [Tracing Representation Progression: Analyzing and Enhancing Layer-Wise Similarity](https://mlsanity.com/paper/2406.14479/) (2024)
- [Fine-Tuned Transformers Show Clusters of Similar Representations Across Layers](https://mlsanity.com/paper/2109.08406/) (2021)
- [Do Large Language Models Perform Latent Multi-Hop Reasoning without Exploiting Shortcuts?](https://mlsanity.com/paper/2411.16679/) (2024)

## Built on by (most cited first, 3 of 3)

- [The Latent Space: Foundation, Evolution, Mechanism, Ability, and Outlook](https://mlsanity.com/paper/2604.02029/) (2026)
- [Think Fast: Estimating No-CoT Task-Completion Time Horizons of Frontier AI Models](https://mlsanity.com/paper/2606.07157/) (2026)
- [Deeper is Not Always Better: Mitigating the Alignment Tax via Confident Layer Decoding](https://mlsanity.com/paper/2606.21906/) (2026)

## Abstract (arXiv)

Large language models (LLMs) perform well on multi-hop reasoning, yet how they internally compose multiple facts remains unclear. Recent work proposes hop-aligned circuit hypothesis, suggesting that bridge entities are computed sequentially across layers before later-hop answers. Through systematic analyses on real-world multi-hop queries, we show that this hop-aligned assumption does not generalize: later-hop answer entities can become decodable earlier than bridge entities, a phenomenon we call layer-order inversion, which strengthens with total hops. To explain this behavior, we propose a probabilistic recall-and-extract framework that models multi-hop reasoning as broad probabilistic recall in shallow MLP layers followed by selective extraction in deeper attention layers. This framework is empirically validated through systematic probing analyses, reinterpreting prior layer-wise decoding evidence, explaining chain-of-thought gains, and providing a mechanistic diagnosis of multi-hop failures despite correct single-hop knowledge. Code is available at https://github.com/laquabe/Layer-Order-Inversion.