mlsanityData to 7 Oct 2026

Multi-Hop Knowledge Composition is Bound by Pretraining Exposure

arXiv 2606.09338 ↗First posted 8 Jun 2026cs.CLcited by 0

On the map: LLM reasoning → Knowledge- and tool-augmented reasoning

Abstract (arXiv)

Large Language Models fail at implicit multi-hop reasoning: a model answers "When was X born?" and "Who is Y's closest friend?" correctly but fails on "When was Y's closest friend born?" in a single forward pass, even when both facts are perfectly memorized and individually retrievable. We study this failure in a controlled natural language setting with a strict separation between individuals exposed to compositional contexts during pretraining and those that never appear in any such context. We confirm that compositional failure persists even at 97% 1-hop accuracy, establishing the gap as a pretraining failure rather than a knowledge absence. We propose and test nine data-centric augmentation formats and find that compositional pretraining transfers to unseen questions for exposed individuals, but never to individuals absent from compositional pretraining, suggesting that exposure to compositional contexts during pretraining is a necessary condition for implicit multi-hop reasoning. Code is available at https://github.com/ykrmm/composition-bound .

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning. Builds on 16 of its 30 references that are in the map. First posted on arXiv 8 Jun 2026.

Builds on

16 of 30 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Do Large Language Models Perform Latent Multi-Hop Reasoning without Exploiting Shortcuts?25 Nov
  2. 2024Do Large Language Models Latently Perform Multi-Hop Reasoning?26 Feb

Other references · 14

  1. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  2. 2019Language Models as Knowledge Bases?cited by 1,922
  3. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
  4. 2023Unifying Large Language Models and Knowledge Graphs: A Roadmapcited by 663
  5. 2023Physics of Language Models: Part 3.1, Knowledge Storage and Extractioncited by 264
  6. 2024Distilling System 2 into System 1cited by 152
  7. 2023Physics of Language Models: Part 3.2, Knowledge Manipulationcited by 142
  8. 2024Hopping Too Late: Exploring the Limitations of Large Language Models on Multi-Hop Queriescited by 78
  9. 2023Towards a Mechanistic Interpretation of Multi-Step Reasoning Capabilities of Language Modelscited by 50
  10. 2024Understanding and Patching Compositional Reasoning in LLMscited by 45
  11. 2024Do Large Language Models Have Compositional Ability? An Investigation into Limitations and Scalabilitycited by 39
  12. 2024Towards a Theoretical Understanding of the 'Reversal Curse' via Training Dynamicscited by 27
2 more
  1. 2025RLP: Reinforcement as a Pretraining Objectivecited by 23
  2. 2025Language models can learn implicit multi-hop reasoning, but only if they have lots of training datacited by 11

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Composition Collapse: Stable Factual Knowledge Does Not Imply Compositional Reasoning
  2. 2026Lasting Effects of Abstract Pretraining Beyond Perplexity
  3. 2025How do Transformers Learn Implicit Reasoning?
  4. 2024Distributional reasoning in LLMs: Parallel reasoning processes in multi-hop reasoning
  5. 2024Lessons from Studying Two-Hop Latent Reasoning
  6. 2026REALHOP: Rethinking Multi-Hop Reasoning Evaluation via Behavioral Auditing
  7. 2026Track, Rank, Crack: Epistemic Working Memory Scales Multi-Hop Reasoning in Language Agents
  8. 2024Lost in the Middle, and In-Between: Enhancing Language Models' Ability to Reason Over Long Contexts in Multi-Hop QA