mlsanityData to 14 Oct 2026

Compositionality decomposed: how do neural networks generalise?

arXiv 1908.08351 ↗First posted 22 Aug 2019cs.CL cs.AI cs.LGcited by 265

Not on the map. Why?

Abstract (arXiv)

Despite a multitude of empirical studies, little consensus exists on whether neural networks are able to generalise compositionally, a controversy that, in part, stems from a lack of agreement about what it means for a neural model to be compositional. As a response to this controversy, we present a set of tests that provide a bridge between, on the one hand, the vast amount of linguistic and philosophical theory about compositionality of language and, on the other, the successful neural models of language. We collect different interpretations of compositionality and translate them into five theoretically grounded tests for models that are formulated on a task-independent level. In particular, we provide tests to investigate (i) if models systematically recombine known parts and rules (ii) if models can extend their predictions beyond the length they have seen in the training data (iii) if models' composition operations are local or global (iv) if models' predictions are robust to synonym substitutions and (v) if models favour rules or exceptions during training. To demonstrate the usefulness of this evaluation paradigm, we instantiate these five tests on a highly compositional data set which we dub PCFG SET and apply the resulting tests to three popular sequence-to-sequence models: a recurrent, a convolution-based and a transformer model. We provide an in-depth analysis of the results, which uncover the strengths and weaknesses of these three architectures and point to potential areas of improvement.

Key facts. Cited by 265 papers in the map (data through 14 Oct 2026). First posted on arXiv 22 Aug 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 32; 2022: 34; 2023: 56; 2024: 58; 2025: 46; 2026: 39.

265 papers.

Sort:

By area:

  1. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  2. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,075
  3. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  4. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 327
  5. 2023SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionalitycited by 243
  6. 2023Large Language Models as General Pattern Machinescited by 197
  7. 2022CREPE: Can Vision-Language Foundation Models Reason Compositionally?cited by 181
  8. 2021A Survey of Zero-shot Generalisation in Deep Reinforcement Learningcited by 170
  9. 2021How much do language models copy from their training data? Evaluating linguistic novelty in text generation using RAVENcited by 124
  10. 2021The Devil is in the Detail: Simple Tricks Improve Systematic Generalization of Transformerscited by 117
  11. 2022Unveiling Transformers with LEGO: a synthetic reasoning taskcited by 106
  12. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 98
38 more
  1. 2023Compositional Abilities Emerge Multiplicatively: Exploring Diffusion Models on a Synthetic Taskcited by 93
  2. 2021Survey of Low-Resource Machine Translationcited by 92
  3. 2023A Theory of Emergent In-Context Learning as Implicit Structure Inductioncited by 86
  4. 2024Transformers Can Achieve Length Generalization But Not Robustlycited by 86
  5. 2023Modular Deep Learningcited by 83
  6. 2022Can Wikipedia Help Offline Reinforcement Learning?cited by 77
  7. 2024No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performancecited by 75
  8. 2023Generalization on the Unseen, Logic Reasoning and Degree Curriculumcited by 73
  9. 2021Making Transformers Solve Compositional Taskscited by 72
  10. 2022Learning to Compose Soft Prompts for Compositional Zero-Shot Learningcited by 67
  11. 2021The Neural Data Router: Adaptive Control Flow in Transformers Improves Systematic Generalizationcited by 62
  12. 2024ConceptMix: A Compositional Image Generation Benchmark with Controllable Difficultycited by 61
  13. 2021The paradox of the compositionality of natural language: a neural machine translation case studycited by 59
  14. 2023Linear Spaces of Meanings: Compositional Structures in Vision-Language Modelscited by 48
  15. 2023Break It Down: Evidence for Structural Compositionality in Neural Networkscited by 44
  16. 2022A Benchmark for Compositional Visual Reasoningcited by 43
  17. 2022Can language models handle recursively nested grammatical structures? A case study on comparing models and humanscited by 41
  18. 2021Disentangling Syntax and Semantics in the Brain with Deep Networkscited by 40
  19. 2021Systematic Generalization with Edge Transformerscited by 38
  20. 2024How Far Can Transformers Reason? The Globality Barrier and Inductive Scratchpadcited by 36
  21. 2021From Machine Learning to Robotics: Challenges and Opportunities for Embodied Intelligencecited by 35
  22. 2023Sparse Universal Transformercited by 32
  23. 2024Reasoning Abilities of Large Language Models: In-Depth Analysis on the Abstraction and Reasoning Corpuscited by 32
  24. 2025Self-Improving Transformers Overcome Easy-to-Hard and Length Generalization Challengescited by 31
  25. 2025CLIP Behaves like a Bag-of-Words Model Cross-modally but not Uni-modallycited by 31
  26. 2022WOODS: Benchmarks for Out-of-Distribution Generalization in Time Seriescited by 30
  27. 2023Learning to Reason and Memorize with Self-Notescited by 30
  28. 2023Beyond Task Performance: Evaluating and Reducing the Flaws of Large Multimodal Models with In-Context Learningcited by 30
  29. 2021How BPE Affects Memorization in Transformerscited by 29
  30. 2023Dissociating language and thought in large language modelscited by 29
  31. 2022Combining Modular Skills in Multitask Learningcited by 28
  32. 2021Symbolic Brittleness in Sequence Models: on Systematic Generalization in Symbolic Mathematicscited by 27
  33. 2023Compositional Sculpting of Iterative Generative Processescited by 27
  34. 2023Approximating Two-Layer Feedforward Networks for Efficient Transformerscited by 27
  35. 2024Universal Length Generalization with Turing Programscited by 25
  36. 2022Structural generalization is hard for sequence-to-sequence modelscited by 24
  37. 2022Language model acceptability judgements are not always robust to contextcited by 22
  38. 2023SLOG: A Structural Generalization Benchmark for Semantic Parsingcited by 22

50 most cited of 265 All 265 (JSON)