# Unveiling the Mechanisms of Multi-Hop Reasoning in Transformers via Identity Bridge

arXiv: https://arxiv.org/abs/2509.24653 · HTML: https://mlsanity.com/paper/2509.24653/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning.
- Builds on 13 of its 34 references that are in the map.
- First posted on arXiv 29 Sep 2025.

## Placed in

- LLM reasoning / [Knowledge- and tool-augmented reasoning](https://mlsanity.com/lr/knowledge-and-tool-augmented-reasoning/)

## Builds on (most cited first, 13 of 13)

- [Measuring and Narrowing the Compositionality Gap in Language Models](https://mlsanity.com/paper/2210.03350/) (2022)
- [2 OLMo 2 Furious](https://mlsanity.com/paper/2501.00656/) (2024)
- [Gradient Descent Maximizes the Margin of Homogeneous Neural Networks](https://mlsanity.com/paper/1906.05890/) (2019)
- [How to use and interpret activation patching](https://mlsanity.com/paper/2404.15255/) (2024)
- [Do Large Language Models Latently Perform Multi-Hop Reasoning?](https://mlsanity.com/paper/2402.16837/) (2024)
- [Break the Chain: Large Language Models Can be Shortcut Reasoners](https://mlsanity.com/paper/2406.06580/) (2024)
- [Interpreting Key Mechanisms of Factual Recall in Transformer-Based Language Models](https://mlsanity.com/paper/2403.19521/) (2024)
- [Complexity Control Facilitates Reasoning-Based Compositional Generalization in Transformers](https://mlsanity.com/paper/2501.08537/) (2025)
- [Lessons from Studying Two-Hop Latent Reasoning](https://mlsanity.com/paper/2411.16353/) (2024)
- [Understanding Finetuning for Factual Knowledge Extraction from Language Models](https://mlsanity.com/paper/2301.11293/) (2023)
- [An Analysis for Reasoning Bias of Language Models with Small Initialization](https://mlsanity.com/paper/2502.04375/) (2025)
- [Generalization or Hallucination? Understanding Out-of-Context Reasoning in Transformers](https://mlsanity.com/paper/2506.10887/) (2025)
- [An Investigation of LLMs' Inefficacy in Understanding Converse Relations](https://mlsanity.com/paper/2310.05163/) (2023)

## Built on by (most cited first, 4 of 4)

- [The Latent Space: Foundation, Evolution, Mechanism, Ability, and Outlook](https://mlsanity.com/paper/2604.02029/) (2026)
- [Breaking the Reversal Curse in Autoregressive Language Models via Identity Bridge](https://mlsanity.com/paper/2602.02470/) (2026)
- [Is Grokking Worthwhile? Functional Analysis and Transferability of Generalization Circuits in Transformers](https://mlsanity.com/paper/2601.09049/) (2026)
- [Feature Resemblance: Towards a Theoretical Understanding of Analogical Reasoning in Transformers](https://mlsanity.com/paper/2603.05143/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) excel at multi-hop reasoning in distribution, yet fail on unseen compositions, a phenomenon known as the curse of two-hop reasoning. In this work, we argue that this phenomenon can be attributed to a missing supervision on the bridge entity. We formalize this gap by introducing identity bridge, a minimal supervision that enforces a identity mapping on bridge tokens. Under this supervision, even a one-layer transformer with uniform attention (Emb-MLP) can achieve out-of-distribution (OOD) two-hop generalization. We provide a theoretical analysis demonstrating that identity bridge induces an implicit regularization effect, leading the model to establish a direct subject-to-answer association. From an empirical perspective, the performance of standard GPT-2 models aligns closely with simple Emb--MLP models across varying levels of problem complexity. Finally, analyses of fine-tuned mainstream LLMs indicate that correct two-hop predictions consistently coincide with the establishment of a subject-to-answer relationship, extending our findings to realistic settings.