Composition Collapse: Stable Factual Knowledge Does Not Imply Compositional Reasoning
On the map: LLM reasoning → Test-time compute and efficient thinking
Abstract (arXiv)
Post-training is routinely evaluated through aggregate benchmark scores that treat multi-hop reasoning as a single capability -- as if a model that answers more questions correctly must be better at assembling facts. We show that this assumption can be misleading: recipes with statistically indistinguishable atomic knowledge produce composition behaviour separated by over 40 percentage points, a phenomenon we call composition collapse: the systematic failure to assemble stably-known facts into chains, invisible to aggregate metrics. We introduce a double-gate protocol that changes the estimand from an aggregate compositionality gap to residual composition failure conditioned on stable atomic access, decomposing post-training gains into three independent channels: atomic stability, residual composition, and critical depth. On a benchmark of temporal factual chains spanning depths 2--11 across four post-training recipes, this decomposition reveals that post-training objectives shift composition capability in directions that aggregate metrics mask, and suggests that claims about multi-hop reasoning improvement should be accompanied by atomic-gate-controlled composition metrics. Diagnostic probes further show that a substantial share of measured composition failure reflects generation-time computation constraints rather than permanent inability to compose.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 9 of its 19 references that are in the map. First posted on arXiv 26 May 2026.
Builds on
9 of 19 references.
References · 9
- 2022Training language models to follow instructions with human feedbackcited by 15,971
- 2024The Llama 3 Herd of Modelscited by 14,715
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
- 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,075
- 2023Faith and Fate: Limits of Transformers on Compositionalitycited by 503
- 2023Physics of Language Models: Part 3.1, Knowledge Storage and Extractioncited by 270
- 2024Hopping Too Late: Exploring the Limitations of Large Language Models on Multi-Hop Queriescited by 78
- 2024Do Large Language Models Perform Latent Multi-Hop Reasoning without Exploiting Shortcuts?cited by 28
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026Multi-Hop Knowledge Composition is Bound by Pretraining Exposure
- 2026Wrong Prediction, Right Answer: Recovering Evidence from Collapsed LLM Sequence Scores
- 2026The Ignition Is Real, and It Lives at the Readout: Latent composition, difficulty-clocked ignition, and the interface-constituted commit in a recurrent-depth reasoner
- 2024Understanding and Patching Compositional Reasoning in LLMs
- 2026Facts Don't Speak Louder than Words: The Behavioral Essence of Long-CoT Distillation
- 2026Track, Rank, Crack: Epistemic Working Memory Scales Multi-Hop Reasoning in Language Agents
- 2025Learning Composable Chains-of-Thought
- 2025Compositional Generalization from Learned Skills via CoT Training: A Theoretical and Structural Analysis for Reasoning