mlsanityData to 14 Oct 2026

Measuring Compositional Generalization: A Comprehensive Method on Realistic Data

arXiv 1912.09713 ↗First posted 20 Dec 2019cs.LG cs.CL stat.MLcited by 284

Not on the map. Why?

Cited by 26 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

State-of-the-art machine learning methods exhibit limited compositional generalization. At the same time, there is a lack of realistic benchmarks that comprehensively measure this ability, which makes it challenging to find and evaluate improvements. We introduce a novel method to systematically construct such benchmarks by maximizing compound divergence while guaranteeing a small atom divergence between train and test sets, and we quantitatively compare this method to other approaches for creating compositional generalization benchmarks. We present a large and realistic natural language question answering dataset that is constructed according to this method, and we use it to analyze the compositional generalization ability of three machine learning architectures. We find that they fail to generalize compositionally and that there is a surprisingly strong negative correlation between compound divergence and accuracy. We also demonstrate how our method can be used to create new compositionality benchmarks on top of the existing SCAN dataset, which confirms these findings.

Key facts. Cited by 284 papers in the map (data through 14 Oct 2026). First posted on arXiv 20 Dec 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 56; 2022: 53; 2023: 69; 2024: 38; 2025: 28; 2026: 40.

284 papers.

Sort:

By area:

  1. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,407
  2. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,075
  3. 2022Towards Reasoning in Large Language Models: A Surveycited by 596
  4. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  5. 2023Augmented Language Models: a Surveycited by 403
  6. 2023Composer: Creative and Controllable Image Synthesis with Composable Conditionscited by 304
  7. 2022Prompting GPT-3 To Be Reliablecited by 260
  8. 2022CREPE: Can Vision-Language Foundation Models Reason Compositionally?cited by 181
  9. 2021A Survey of Zero-shot Generalisation in Deep Reinforcement Learningcited by 170
  10. 2022Diverse Demonstrations Improve In-context Compositional Generalizationcited by 139
  11. 2021How much do language models copy from their training data? Evaluating linguistic novelty in text generation using RAVENcited by 124
  12. 2021AGQA: A Benchmark for Compositional Spatio-Temporal Reasoningcited by 120
38 more
  1. 2021The Devil is in the Detail: Simple Tricks Improve Systematic Generalization of Transformerscited by 117
  2. 2023Randomized Positional Encodings Boost Length Generalization of Transformerscited by 117
  3. 2021Case-based Reasoning for Natural Language Queries over Knowledge Basescited by 107
  4. 2021KaggleDBQA: Realistic Evaluation of Text-to-SQL Parserscited by 106
  5. 2022Unveiling Transformers with LEGO: a synthetic reasoning taskcited by 106
  6. 2022What Artificial Neural Networks Can Tell Us About Human Language Acquisitioncited by 99
  7. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 98
  8. 2024Easy-to-Hard Generalization: Scalable Alignment Beyond Human Supervisioncited by 96
  9. 2022Compositional Semantic Parsing with Large Language Modelscited by 95
  10. 2021Exploring Underexplored Limitations of Cross-Domain Text-to-SQL Generalizationcited by 92
  11. 2021A Survey on Complex Knowledge Base Question Answering: Methods, Challenges and Solutionscited by 78
  12. 2021Making Transformers Solve Compositional Taskscited by 72
  13. 2023Compositional Generalization from First Principlescited by 64
  14. 2024Demystifying Chains, Trees, and Graphs of Thoughtscited by 63
  15. 2021Meta-Learning to Compositionally Generalizecited by 62
  16. 2021The Neural Data Router: Adaptive Control Flow in Transformers Improves Systematic Generalizationcited by 62
  17. 2024ConceptMix: A Compositional Image Generation Benchmark with Controllable Difficultycited by 61
  18. 2021The paradox of the compositionality of natural language: a neural machine translation case studycited by 59
  19. 2022TIARA: Multi-grained Retrieval for Robust Question Answering over Large Knowledge Basescited by 59
  20. 2023Coverage-based Example Selection for In-Context Learningcited by 59
  21. 2022UnifiedQA-v2: Stronger Generalization via Broader Cross-Format Trainingcited by 57
  22. 2021Unlocking Compositional Generalization in Pre-trained Models Using Intermediate Representationscited by 55
  23. 2021Complex Knowledge Base Question Answering: A Surveycited by 52
  24. 2023How Do In-Context Examples Affect Compositional Generalization?cited by 51
  25. 2022Evaluating the Impact of Model Scale for Compositional Generalization in Semantic Parsingcited by 50
  26. 2025OMEGA: Can LLMs Reason Outside the Box in Math? Evaluating Exploratory, Compositional, and Transformative Generalizationcited by 50
  27. 2021Improving Compositional Generalization with Latent Structure and Data Augmentationcited by 46
  28. 2024Top Leaderboard Ranking = Top Coding Proficiency, Always? EvoEval: Evolving Coding Benchmarks via LLMcited by 40
  29. 2022QALD-9-plus: A Multilingual Dataset for Question Answering over DBpedia and Wikidata Translated by Native Speakerscited by 39
  30. 2021Systematic Generalization with Edge Transformerscited by 38
  31. 2023The Impact of Depth on Compositional Generalization in Transformer Language Modelscited by 38
  32. 2024Efficient Data Collection for Robotic Manipulation via Compositional Generalizationcited by 37
  33. 2021On Compositional Generalization of Neural Machine Translationcited by 34
  34. 2022Modern Baselines for SPARQL Semantic Parsingcited by 34
  35. 2023Skills-in-Context Prompting: Unlocking Compositionality in Large Language Modelscited by 33
  36. 2023Sparse Universal Transformercited by 32
  37. 2021Learning Algebraic Recombination for Compositional Generalizationcited by 31
  38. 2022Do Vision-Language Pretrained Models Learn Composable Primitive Concepts?cited by 31

50 most cited of 284 All 284 (JSON)