mlsanityData to 14 Oct 2026

Measuring and Narrowing the Compositionality Gap in Language Models

arXiv 2210.03350 ↗First posted 7 Oct 2022cs.CLcited by 1,075

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

We investigate the ability of language models to perform compositional reasoning tasks where the overall solution depends on correctly composing the answers to sub-problems. We measure how often models can correctly answer all sub-problems but not generate the overall solution, a ratio we call the compositionality gap. We evaluate this ratio by asking multi-hop questions with answers that require composing multiple facts unlikely to have been observed together during pretraining. In the GPT-3 family of models, as model size increases we show that the single-hop question answering performance improves faster than the multi-hop performance does, therefore the compositionality gap does not decrease. This surprising result suggests that while more powerful models memorize and recall more factual knowledge, they show no corresponding improvement in their ability to perform this kind of compositional reasoning. We then demonstrate how elicitive prompting (such as chain of thought) narrows the compositionality gap by reasoning explicitly. We present a new method, self-ask, that further improves on chain of thought. In our method, the model explicitly asks itself (and answers) follow-up questions before answering the initial question. We finally show that self-ask's structured prompting lets us easily plug in a search engine to answer the follow-up questions, which additionally improves accuracy.

Key facts. Cited by 1,075 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 56 of its 66 references that are in the map. First posted on arXiv 7 Oct 2022.

Builds on

56 of 66 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Models6 Oct
  2. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Models30 Nov

Other references · 54

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  4. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  5. 2022Emergent Abilities of Large Language Modelscited by 2,512
  6. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  7. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  8. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  9. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,407
  10. 2021Improving language models by retrieving from trillions of tokenscited by 1,293
  11. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,282
  12. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
38 more
  1. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,210
  2. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,077
  3. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 916
  4. 2018Personalizing Dialogue Agents: I have a dog, do you have pets too?cited by 828
  5. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 800
  6. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 719
  7. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 650
  8. 2020Recipes for building an open-domain chatbotcited by 593
  9. 2022Decomposed Prompting: A Modular Approach for Solving Complex Taskscited by 558
  10. 2016Adaptive Computation Time for Recurrent Neural Networkscited by 506
  11. 2020Towards a Human-like Open-Domain Chatbotcited by 470
  12. 2018The Web as a Knowledge-base for Answering Complex Questionscited by 410
  13. 2017SkipNet: Learning Dynamic Routing in Convolutional Networkscited by 332
  14. 2022Teaching language models to support answers with verified quotescited by 292
  15. 2019Measuring Compositional Generalization: A Comprehensive Method on Realistic Datacited by 284
  16. 2019Compositionality decomposed: how do neural networks generalise?cited by 265
  17. 2022RealTime QA: What's the Answer Right Now?cited by 244
  18. 2021Generated Knowledge Prompting for Commonsense Reasoningcited by 214
  19. 2015A Neural Conversational Modelcited by 196
  20. 2019The Second Conversational Intelligence Challenge (ConvAI2)cited by 179
  21. 2021Reframing Instructional Prompts to GPTk's Languagecited by 176
  22. 2017Adaptive Neural Networks for Efficient Inferencecited by 163
  23. 2019oLMpics -- On what Language Model Pre-training Capturescited by 161
  24. 2020Unsupervised Commonsense Question Answering with Self-Talkcited by 161
  25. 2019Neural Logic Machinescited by 145
  26. 2021Not All Images are Worth 16x16 Words: Dynamic Transformers for Efficient Image Recognitioncited by 144
  27. 2020Break It Down: A Question Understanding Benchmarkcited by 131
  28. 2020The Right Tool for the Job: Matching Model and Instance Complexitiescited by 126
  29. 2015Neural Responding Machine for Short-Text Conversationcited by 124
  30. 2019Multi-hop Reading Comprehension through Question Decomposition and Rescoringcited by 119
  31. 2020Unsupervised Question Decomposition for Question Answeringcited by 118
  32. 2021Can You Learn an Algorithm? Generalizing from Easy to Hard Problems with Recurrent Networkscited by 117
  33. 2015A Neural Network Approach to Context-Sensitive Generation of Conversational Responsescited by 98
  34. 2020Text Modular Networks: Learning to Decompose Tasks in the Language of Existing Modelscited by 65
  35. 2019Answering Complex Open-domain Questions Through Iterative Query Generationcited by 60
  36. 2020Knowledge-Aware Language Model Pretrainingcited by 52
  37. 2017Ask the Right Questions: Active Question Reformulation with Reinforcement Learningcited by 44
  38. 2019Answer-based Adversarial Training for Generating Clarification Questionscited by 43

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 16; 2023: 176; 2024: 176; 2025: 322; 2026: 385.

1,075 papers.

Sort:

By area:

  1. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  2. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  3. 2023A Survey of Large Language Modelscited by 2,576
  4. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
  5. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,444
  6. 2022Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questionscited by 923
  7. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 841
  8. 2023DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelinescited by 832
  9. 2023Large Language Modelscited by 780
  10. 2022A Survey on In-context Learningcited by 721
  11. 2022PAL: Program-aided Language Modelscited by 652
  12. 2023ChemCrow: Augmenting large-language models with chemistry toolscited by 631
38 more
  1. 2022Towards Reasoning in Large Language Models: A Surveycited by 596
  2. 2023Enabling Large Language Models to Generate Text with Citationscited by 595
  3. 2023Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Modelscited by 560
  4. 2023Active Retrieval Augmented Generationcited by 516
  5. 2025Group-in-Group Policy Optimization for LLM Agent Trainingcited by 497
  6. 2024Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexitycited by 451
  7. 2025Search-o1: Agentic Search-Enhanced Large Reasoning Modelscited by 433
  8. 2023TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answeringcited by 428
  9. 2023The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"cited by 407
  10. 2023Augmented Language Models: a Surveycited by 403
  11. 2025Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscationcited by 384
  12. 2023Language Models can Solve Computer Taskscited by 372
  13. 2023Making Retrieval-Augmented Language Models Robust to Irrelevant Contextcited by 369
  14. 2025ToolRL: Reward is All Tool Learning Needscited by 362
  15. 2023Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergycited by 346
  16. 2025R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learningcited by 336
  17. 2023Large Language Models for Information Retrieval: A Surveycited by 331
  18. 2024Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Modelscited by 326
  19. 2023Chain-of-Verification Reduces Hallucination in Large Language Modelscited by 324
  20. 2022Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLPcited by 318
  21. 2023FreshLLMs: Refreshing Large Language Models with Search Engine Augmentationcited by 318
  22. 2023How Language Model Hallucinations Can Snowballcited by 311
  23. 2023Tool Learning with Foundation Modelscited by 302
  24. 2023MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop Questionscited by 300
  25. 2025DeepResearcher: Scaling Deep Research via Reinforcement Learning in Real-world Environmentscited by 286
  26. 2024HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Modelscited by 281
  27. 2026SkillRL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learningcited by 278
  28. 2024TextGrad: Automatic "Differentiation" via Textcited by 273
  29. 2023FireAct: Toward Language Agent Fine-tuningcited by 256
  30. 2023Challenges and Applications of Large Language Modelscited by 254
  31. 2022Reasoning with Language Model Prompting: A Surveycited by 252
  32. 2023FacTool: Factuality Detection in Generative AI -- A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarioscited by 251
  33. 2025MEM1: Learning to Synergize Memory and Reasoning for Efficient Long-Horizon Agentscited by 238
  34. 2022Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matterscited by 222
  35. 2023Query Rewriting for Retrieval-Augmented Large Language Modelscited by 213
  36. 2025ZeroSearch: Incentivize the Search Capability of LLMs without Searchingcited by 208
  37. 2023What Algorithms can Transformers Learn? A Study in Length Generalizationcited by 199
  38. 2023ART: Automatic multi-step reasoning and tool-use for large language modelscited by 197

50 most cited of 1,075 All 1,075 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Distributional reasoning in LLMs: Parallel reasoning processes in multi-hop reasoning
  2. 2024Do Large Language Models Have Compositional Ability? An Investigation into Limitations and Scalability
  3. 2025Answer-Centric or Reasoning-Driven? Uncovering the Latent Memory Anchor in LLMs
  4. 2026Compositional Reasoning in Language Models under Reinforcement Learning Post-Training
  5. 2024Lost in the Middle, and In-Between: Enhancing Language Models' Ability to Reason Over Long Contexts in Multi-Hop QA
  6. 2024Understanding and Patching Compositional Reasoning in LLMs
  7. 2023Teaching Smaller Language Models To Generalise To Unseen Compositional Questions
  8. 2024Exploring the Compositional Deficiency of Large Language Models in Mathematical Reasoning