mlsanityData to 7 Oct 2026

Making Transformers Solve Compositional Tasks

arXiv 2108.04378 ↗First posted 9 Aug 2021cs.AI cs.CLcited by 72

Not on the map. Why?

Abstract (arXiv)

Several studies have reported the inability of Transformer models to generalize compositionally, a key type of generalization in many NLP tasks such as semantic parsing. In this paper we explore the design space of Transformer models showing that the inductive biases given to the model by several design decisions significantly impact compositional generalization. Through this exploration, we identified Transformer configurations that generalize compositionally significantly better than previously reported in the literature in a diverse set of compositional tasks, and that achieve state-of-the-art results in a semantic parsing compositional generalization benchmark (COGS), and a string edit operation composition benchmark (PCFG).

Key facts. Cited by 72 papers in the map (data through 7 Oct 2026). Builds on 24 of its 34 references that are in the map. First posted on arXiv 9 Aug 2021.

Builds on

24 of 34 references.

References · 24

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2020Longformer: The Long-Document Transformercited by 3,362
  5. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  6. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  7. 2020Shortcut Learning in Deep Neural Networkscited by 1,928
  8. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  9. 2020Reformer: The Efficient Transformercited by 1,658
  10. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  11. 2018Self-Attention with Relative Position Representationscited by 1,112
  12. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
12 more
  1. 2019Measuring Compositional Generalization: A Comprehensive Method on Realistic Datacited by 280
  2. 2019Compositionality decomposed: how do neural networks generalise?cited by 264
  3. 2020COGS: A Compositional Generalization Challenge Based on Semantic Interpretationcited by 260
  4. 2019Good-Enough Compositional Data Augmentationcited by 158
  5. 2019Adaptive Attention Span in Transformerscited by 148
  6. 2019HIBERT: Document Level Pre-training of Hierarchical Bidirectional Transformers for Document Summarizationcited by 120
  7. 2020Compositional Generalization in Semantic Parsing: Pre-training vs. Specialized Architecturescited by 95
  8. 2019Compositional generalization in a deep seq2seq model by separating syntax and semanticscited by 56
  9. 2021Unlocking Compositional Generalization in Pre-trained Models Using Intermediate Representationscited by 55
  10. 2020Compositional Generalization by Learning Analytical Expressionscited by 53
  11. 2018Memorize or generalize? Searching for a compositional RNN in a haystackcited by 41
  12. 2021Improving Compositional Generalization in Classification Tasks via Structure Annotationscited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 7; 2022: 15; 2023: 16; 2024: 17; 2025: 12; 2026: 5.

72 papers.

Sort:

By area:

  1. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 319
  2. 2023Teaching Arithmetic to Small Transformerscited by 121
  3. 2021The Devil is in the Detail: Simple Tricks Improve Systematic Generalization of Transformerscited by 117
  4. 2023Randomized Positional Encodings Boost Length Generalization of Transformerscited by 114
  5. 2022Fusing finetuned models for better pretrainingcited by 110
  6. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 97
  7. 2021The Neural Data Router: Adaptive Control Flow in Transformers Improves Systematic Generalizationcited by 62
  8. 2024MoEUT: Mixture-of-Experts Universal Transformerscited by 55
  9. 2022Evaluating the Impact of Model Scale for Compositional Generalization in Semantic Parsingcited by 50
  10. 2021Improving Compositional Generalization with Latent Structure and Data Augmentationcited by 46
  11. 2021Systematic Generalization with Edge Transformerscited by 38
  12. 2023The Impact of Depth on Compositional Generalization in Transformer Language Modelscited by 37
38 more
  1. 2021The Grammar-Learning Trajectories of Neural Language Modelscited by 36
  2. 2024How Far Can Transformers Reason? The Globality Barrier and Inductive Scratchpadcited by 35
  3. 2022Uncontrolled Lexical Exposure Leads to Overestimation of Compositional Generalization in Pretrained Modelscited by 26
  4. 2023Neural-Logic Human-Object Interaction Detectioncited by 25
  5. 2022Structural generalization is hard for sequence-to-sequence modelscited by 24
  6. 2023ReCOGS: How Incidental Details of a Logical Form Overshadow an Evaluation of Semantic Interpretationcited by 24
  7. 2024A Philosophical Introduction to Language Models -- Part I: Continuity With Classic Debatescited by 22
  8. 2023Compositional Capabilities of Autoregressive Transformers: A Study on Synthetic, Interpretable Taskscited by 21
  9. 2022Revisiting the Compositional Generalization Abilities of Neural Sequence Modelscited by 20
  10. 2022Systematic Generalization and Emergent Structures in Transformers Trained on Structured Taskscited by 19
  11. 2022The Curious Case of Absolute Position Embeddingscited by 17
  12. 2023ExeDec: Execution Decomposition for Compositional Generalization in Neural Program Synthesiscited by 17
  13. 2024A Survey on Compositional Learning of AI Models: Theoretical and Experimental Practicescited by 15
  14. 2023Sequential Query Encoding For Complex Query Answering on Knowledge Graphscited by 13
  15. 2022LogicInference: A New Dataset for Teaching Logical Inference to seq2seq Modelscited by 12
  16. 2022Neural-Symbolic Recursive Machine for Systematic Generalizationcited by 12
  17. 2025Strassen Attention, Split VC Dimension and Compositionality in Transformerscited by 12
  18. 2024AlgoFormer: An Efficient Transformer Framework with Algorithmic Structurescited by 11
  19. 2024Benchmarking GPT-4 on Algorithmic Problems: A Systematic Evaluation of Prompting Strategiescited by 9
  20. 2023Learning to Substitute Spans towards Improving Compositional Generalizationcited by 8
  21. 2025The Role of Sparsity for Length Generalization in Transformerscited by 8
  22. 2025NePTune: A Neuro-Pythonic Framework for Tunable Compositional Reasoning on Vision-Languagecited by 8
  23. 2021LAGr: Labeling Aligned Graphs for Improving Systematic Generalization in Semantic Parsingcited by 7
  24. 2024ALTA: Compiler-Based Analysis of Transformerscited by 7
  25. 2021Iterative Decoding for Compositional Generalization in Transformerscited by 6
  26. 2022Compositional Generalisation with Structured Reordering and Fertility Layerscited by 6
  27. 2025Beyond Statistical Learning: Exact Learning Is Essential for General Intelligencecited by 6
  28. 2024Towards Understanding the Relationship between In-context Learning and Compositional Generalizationcited by 5
  29. 2024What makes Models Compositional? A Theoretical View: With Supplementcited by 5
  30. 2024Philosophy of Cognitive Science in the Age of Deep Learningcited by 5
  31. 2024Quantifying artificial intelligence through algorithmic generalizationcited by 5
  32. 2023MAQA: A Multimodal QA Benchmark for Negationcited by 4
  33. 2023Probabilistic Transformer: A Probabilistic Dependency Model for Contextual Word Representationcited by 4
  34. 2025Decomposing Complex Visual Comprehension into Atomic Visual Skills for Vision Language Modelscited by 4
  35. 2024On the generalization capacity of neural networks during generic multimodal reasoningcited by 3
  36. 2024SPOR: A Comprehensive and Practical Evaluation Method for Compositional Generalization in Data-to-Text Generationcited by 3
  37. 2025Long-Short Alignment for Effective Long-Context Modeling in LLMscited by 3
  38. 2025To Infinity and Beyond: Tool-Use Unlocks Length Generalization in State Space Modelscited by 3

50 most cited of 72 All 72 (JSON)