mlsanityData to 7 Oct 2026

COGS: A Compositional Generalization Challenge Based on Semantic Interpretation

arXiv 2010.05465 ↗First posted 12 Oct 2020cs.CLcited by 260

Not on the map. Why?

Abstract (arXiv)

Natural language is characterized by compositionality: the meaning of a complex expression is constructed from the meanings of its constituent parts. To facilitate the evaluation of the compositional abilities of language processing architectures, we introduce COGS, a semantic parsing dataset based on a fragment of English. The evaluation portion of COGS contains multiple systematic gaps that can only be addressed by compositional generalization; these include new combinations of familiar syntactic structures, or new combinations of familiar words and familiar structures. In experiments with Transformers and LSTMs, we found that in-distribution accuracy on the COGS test set was near-perfect (96--99%), but generalization accuracy was substantially lower (16--35%) and showed high sensitivity to random seed (±6--8%). These findings indicate that contemporary standard NLP models are limited in their compositional generalization capacity, and position COGS as a good way to measure progress.

Key facts. Cited by 260 papers in the map (data through 7 Oct 2026). First posted on arXiv 12 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 36; 2022: 42; 2023: 57; 2024: 44; 2025: 36; 2026: 45.

260 papers.

Sort:

By area:

  1. 2021Dynabench: Rethinking Benchmarking in NLPcited by 447
  2. 2022Prompting GPT-3 To Be Reliablecited by 258
  3. 2021ExT5: Towards Extreme Multi-Task Scaling for Transfer Learningcited by 161
  4. 2024Benchmarking Complex Instruction-Following with Multiple Constraints Compositioncited by 147
  5. 2021How much do language models copy from their training data? Evaluating linguistic novelty in text generation using RAVENcited by 124
  6. 2021The Devil is in the Detail: Simple Tricks Improve Systematic Generalization of Transformerscited by 117
  7. 2023Randomized Positional Encodings Boost Length Generalization of Transformerscited by 114
  8. 2022What Artificial Neural Networks Can Tell Us About Human Language Acquisitioncited by 97
  9. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 97
  10. 2022Compositional Semantic Parsing with Large Language Modelscited by 95
  11. 2023A Theory of Emergent In-Context Learning as Implicit Structure Inductioncited by 86
  12. 2021Making Transformers Solve Compositional Taskscited by 72
38 more
  1. 2024The Semantic Hub Hypothesis: Language Models Share Semantic Representations Across Languages and Modalitiescited by 69
  2. 2021Tailor: Generating and Perturbing Text with Semantic Controlscited by 67
  3. 2024Demystifying Chains, Trees, and Graphs of Thoughtscited by 63
  4. 2021Meta-Learning to Compositionally Generalizecited by 62
  5. 2021Frequency Effects on Syntactic Rule Learning in Transformerscited by 59
  6. 2021The paradox of the compositionality of natural language: a neural machine translation case studycited by 58
  7. 2025CF-VLM:CounterFactual Vision-Language Fine-tuningcited by 57
  8. 2022Linear Connectivity Reveals Generalization Strategiescited by 55
  9. 2023How Do In-Context Examples Affect Compositional Generalization?cited by 51
  10. 2022Evaluating the Impact of Model Scale for Compositional Generalization in Semantic Parsingcited by 50
  11. 2021Improving Compositional Generalization with Latent Structure and Data Augmentationcited by 46
  12. 2023Skill-Based Few-Shot Selection for In-Context Learningcited by 46
  13. 2022Neurocompositional computing: From the Central Paradox of Cognition to a new generation of AI systemscited by 44
  14. 2022Why is Winoground Hard? Investigating Failures in Visuolinguistic Compositionalitycited by 44
  15. 2023Break It Down: Evidence for Structural Compositionality in Neural Networkscited by 44
  16. 2022Characterizing Intrinsic Compositionality in Transformers with Tree Projectionscited by 40
  17. 2024Do Large Language Models Have Compositional Ability? An Investigation into Limitations and Scalabilitycited by 39
  18. 2021Systematic Generalization with Edge Transformerscited by 38
  19. 2023The Impact of Depth on Compositional Generalization in Transformer Language Modelscited by 37
  20. 2025Between Circuits and Chomsky: Pre-pretraining on Formal Languages Imparts Linguistic Biasescited by 36
  21. 2024Efficient Data Collection for Robotic Manipulation via Compositional Generalizationcited by 35
  22. 2024Filtered Corpus Training (FiCT) Shows that Language Models can Generalize from Indirect Evidencecited by 35
  23. 2021Learning Algebraic Recombination for Compositional Generalizationcited by 31
  24. 2021Sequence-to-Sequence Learning with Latent Neural Grammarscited by 30
  25. 2021Disentangled Sequence to Sequence Learning for Compositional Generalizationcited by 30
  26. 2025The Curse of CoT: On the Limitations of Chain-of-Thought in In-Context Learningcited by 30
  27. 2021How BPE Affects Memorization in Transformerscited by 29
  28. 2022Unobserved Local Structures Make Compositional Generalization Hardcited by 29
  29. 2022WOODS: Benchmarks for Out-of-Distribution Generalization in Time Seriescited by 29
  30. 2021Lexicon Learning for Few-Shot Neural Sequence Modelingcited by 28
  31. 2024How Does Code Pretraining Affect Language Model Task Performance?cited by 28
  32. 2021Are Pre-trained Convolutions Better than Pre-trained Transformers?cited by 27
  33. 2021Finding needles in a haystack: Sampling Structurally-diverse Training Sets from Synthetic Data for Compositional Generalizationcited by 27
  34. 2021Symbolic Brittleness in Sequence Models: on Systematic Generalization in Symbolic Mathematicscited by 27
  35. 2022On Neural Architecture Inductive Biases for Relational Taskscited by 26
  36. 2022Uncontrolled Lexical Exposure Leads to Overestimation of Compositional Generalization in Pretrained Modelscited by 26
  37. 2021COVR: A test-bed for Visually Grounded Compositional Generalization with real imagescited by 25
  38. 2023Neural-Logic Human-Object Interaction Detectioncited by 25

50 most cited of 260 All 260 (JSON)