mlsanityData to 7 Oct 2026

The Devil is in the Detail: Simple Tricks Improve Systematic Generalization of Transformers

arXiv 2108.12284 ↗First posted 26 Aug 2021cs.LG cs.AI cs.NEcited by 117

Not on the map. Why?

Abstract (arXiv)

Recently, many datasets have been proposed to test the systematic generalization ability of neural networks. The companion baseline Transformers, typically trained with default hyper-parameters from standard tasks, are shown to fail dramatically. Here we demonstrate that by revisiting model configurations as basic as scaling of embeddings, early stopping, relative positional embedding, and Universal Transformer variants, we can drastically improve the performance of Transformers on systematic generalization. We report improvements on five popular datasets: SCAN, CFQ, PCFG, COGS, and Mathematics dataset. Our models improve accuracy from 50% to 85% on the PCFG productivity split, and from 35% to 81% on COGS. On SCAN, relative positional embedding largely mitigates the EOS decision problem (Newman et al., 2020), yielding 100% accuracy on the length split with a cutoff at 26. Importantly, performance differences between these models are typically invisible on the IID data split. This calls for proper generalization validation sets for developing neural networks that generalize systematically. We publicly release the code to reproduce our results.

Key facts. Cited by 117 papers in the map (data through 7 Oct 2026). Builds on 39 of its 56 references that are in the map. First posted on arXiv 26 Aug 2021.

Builds on

39 of 56 references.

References · 39

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  4. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  5. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  6. 2018Self-Attention with Relative Position Representationscited by 1,112
  7. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
  8. 2018Universal Transformerscited by 613
  9. 2016Adaptive Computation Time for Recurrent Neural Networkscited by 479
  10. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 471
  11. 2019Analysing Mathematical Reasoning Abilities of Neural Modelscited by 331
  12. 2019Measuring Compositional Generalization: A Comprehensive Method on Realistic Datacited by 280
27 more
  1. 2019Compositionality decomposed: how do neural networks generalise?cited by 264
  2. 2020COGS: A Compositional Generalization Challenge Based on Semantic Interpretationcited by 260
  3. 2020On the Binding Problem in Artificial Neural Networkscited by 230
  4. 2019Good-Enough Compositional Data Augmentationcited by 158
  5. 2020Compositional Generalization and Natural Language Variation: Can a Semantic Parsing Approach Handle Both?cited by 147
  6. 2015Neural GPUs Learn Algorithmscited by 118
  7. 2019Compositional generalization through meta sequence-to-sequence learningcited by 98
  8. 2020Are Neural Nets Modular? Inspecting Functional Modularity Through Differentiable Weight Maskscited by 97
  9. 2020Compositional Generalization in Semantic Parsing: Pre-training vs. Specialized Architecturescited by 95
  10. 2021Going Beyond Linear Transformers with Recurrent Fast Weight Programmerscited by 84
  11. 2020Compositional Generalization via Neural-Symbolic Stack Machinescited by 80
  12. 2020Span-based Semantic Parsing for Compositional Generalizationcited by 77
  13. 2020Learning Compositional Rules via Neural Program Synthesiscited by 74
  14. 2021Making Transformers Solve Compositional Taskscited by 72
  15. 2019CLOSURE: Assessing Systematic Generalization of CLEVR Modelscited by 63
  16. 2019Improving Deep Transformer with Depth-Scaled Initialization and Merged Attentioncited by 59
  17. 2019Compositional generalization in a deep seq2seq model by separating syntax and semanticscited by 56
  18. 2021Unlocking Compositional Generalization in Pre-trained Models Using Intermediate Representationscited by 55
  19. 2020Compositional Generalization by Learning Analytical Expressionscited by 53
  20. 2019Compositional Generalization for Primitive Substitutionscited by 48
  21. 2020The EOS Decision and Length Extrapolationcited by 46
  22. 2021GradInit: Learning to Initialize Neural Networks for Stable and Efficient Trainingcited by 44
  23. 2018Memorize or generalize? Searching for a compositional RNN in a haystackcited by 41
  24. 2019Enhancing the Transformer with Explicit Relational Encoding for Math Problem Solvingcited by 34
  25. 2020Hierarchical Poset Decoding for Compositional Generalization in Languagecited by 26
  26. 2019Transcoding compositionally: using attention to find more generalizable solutionscited by 14
  27. 2019Improving Differentiable Neural Computers Through Memory Masking, De-allocation, and Link Distribution Sharpness Controlcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 11; 2022: 28; 2023: 28; 2024: 25; 2025: 15; 2026: 10.

117 papers.

Sort:

By area:

  1. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 319
  2. 2023Randomized Positional Encodings Boost Length Generalization of Transformerscited by 114
  3. 2022Teaching Algorithmic Reasoning via In-context Learningcited by 111
  4. 2022Fusing finetuned models for better pretrainingcited by 110
  5. 2022General-Purpose In-Context Learning by Meta-Learning Transformerscited by 102
  6. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 97
  7. 2023Generalization on the Unseen, Logic Reasoning and Degree Curriculumcited by 73
  8. 2023Efficient Benchmarking of Language Modelscited by 71
  9. 2021The Neural Data Router: Adaptive Control Flow in Transformers Improves Systematic Generalizationcited by 62
  10. 2024MoEUT: Mixture-of-Experts Universal Transformerscited by 55
  11. 2023Grokking of Hierarchical Structure in Vanilla Transformerscited by 53
  12. 2023How Do In-Context Examples Affect Compositional Generalization?cited by 51
38 more
  1. 2022Evaluating the Impact of Model Scale for Compositional Generalization in Semantic Parsingcited by 50
  2. 2021Improving Compositional Generalization with Latent Structure and Data Augmentationcited by 46
  3. 2021Systematic Generalization with Edge Transformerscited by 38
  4. 2023The Impact of Depth on Compositional Generalization in Transformer Language Modelscited by 37
  5. 2024How Far Can Transformers Reason? The Globality Barrier and Inductive Scratchpadcited by 35
  6. 2021Sequence-to-Sequence Learning with Latent Neural Grammarscited by 30
  7. 2022Unobserved Local Structures Make Compositional Generalization Hardcited by 29
  8. 2022WOODS: Benchmarks for Out-of-Distribution Generalization in Time Seriescited by 29
  9. 2023Sparse Universal Transformercited by 28
  10. 2023Towards Autoformalization of Mathematics and Code Correctness: Experiments with Elementary Proofscited by 27
  11. 2022Uncontrolled Lexical Exposure Leads to Overestimation of Compositional Generalization in Pretrained Modelscited by 26
  12. 2022Structural generalization is hard for sequence-to-sequence modelscited by 24
  13. 2023ReCOGS: How Incidental Details of a Logical Form Overshadow an Evaluation of Semantic Interpretationcited by 24
  14. 2023DecoderLens: Layerwise Interpretation of Encoder-Decoder Transformerscited by 22
  15. 2023SLOG: A Structural Generalization Benchmark for Semantic Parsingcited by 22
  16. 2023Compositional Capabilities of Autoregressive Transformers: A Study on Synthetic, Interpretable Taskscited by 21
  17. 2022Revisiting the Compositional Generalization Abilities of Neural Sequence Modelscited by 20
  18. 2021ReaSCAN: Compositional Reasoning in Language Groundingcited by 19
  19. 2022Systematic Generalization and Emergent Structures in Transformers Trained on Structured Taskscited by 19
  20. 2022The Curious Case of Absolute Position Embeddingscited by 17
  21. 2023ExeDec: Execution Decomposition for Compositional Generalization in Neural Program Synthesiscited by 17
  22. 2025RExBench: Can coding agents autonomously implement AI research extensions?cited by 17
  23. 2024A Survey on Compositional Learning of AI Models: Theoretical and Experimental Practicescited by 15
  24. 2024Limits of Transformer Language Models on Learning to Compose Algorithmscited by 13
  25. 2021Compositional Attention: Disentangling Search and Retrievalcited by 12
  26. 2022LogicInference: A New Dataset for Teaching Logical Inference to seq2seq Modelscited by 12
  27. 2022Eliciting and Understanding Cross-Task Skills with Task-Level Mixture-of-Expertscited by 12
  28. 2022Neural-Symbolic Recursive Machine for Systematic Generalizationcited by 12
  29. 2022CTL++: Evaluating Generalization on Never-Seen Compositional Patterns of Known Functions, and Compatibility of Neural Representationscited by 12
  30. 2022When Can Transformers Ground and Compose: Insights from Compositional Generalization Benchmarkscited by 11
  31. 2023Large Language Model Programscited by 11
  32. 2026Hyperloop Transformerscited by 11
  33. 2024Benchmarking GPT-4 on Algorithmic Problems: A Systematic Evaluation of Prompting Strategiescited by 9
  34. 2022Structurally Diverse Sampling for Sample-Efficient Training and Comprehensive Evaluationcited by 8
  35. 2022Benchmarking Long-tail Generalization with Likelihood Splitscited by 8
  36. 2022Mutual Exclusivity Training and Primitive Augmentation to Induce Compositionalitycited by 8
  37. 2025The Role of Sparsity for Length Generalization in Transformerscited by 8
  38. 2021LAGr: Labeling Aligned Graphs for Improving Systematic Generalization in Semantic Parsingcited by 7

50 most cited of 117 All 117 (JSON)