mlsanityData to 7 Oct 2026

Transformers Generalize Linearly

arXiv 2109.12036 ↗First posted 24 Sep 2021cs.CLcited by 19

Not on the map. Why?

Abstract (arXiv)

Natural language exhibits patterns of hierarchically governed dependencies, in which relations between words are sensitive to syntactic structure rather than linear ordering. While re-current network models often fail to generalize in a hierarchically sensitive way (McCoy et al.,2020) when trained on ambiguous data, the improvement in performance of newer Trans-former language models (Vaswani et al., 2017)on a range of syntactic benchmarks trained on large data sets (Goldberg, 2019; Warstadtet al., 2019) opens the question of whether these models might exhibit hierarchical generalization in the face of impoverished data.In this paper we examine patterns of structural generalization for Transformer sequence-to-sequence models and find that not only do Transformers fail to generalize hierarchically across a wide variety of grammatical mapping tasks, but they exhibit an even stronger preference for linear generalization than comparable recurrent networks

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 21 of its 26 references that are in the map. First posted on arXiv 24 Sep 2021.

Builds on

21 of 26 references.

References · 21

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  4. 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,024
  5. 2020A Primer in BERTology: What we know about how BERT workscited by 807
  6. 2019Theoretical Limitations of Self-Attention in Neural Sequence Modelscited by 286
  7. 2020COGS: A Compositional Generalization Challenge Based on Semantic Interpretationcited by 260
  8. 2018Targeted Syntactic Evaluation of Language Modelscited by 239
  9. 2018Colorless green recurrent networks dream hierarchicallycited by 210
  10. 2019Assessing BERT's Syntactic Abilitiescited by 191
  11. 2020A Systematic Assessment of Syntactic Generalization in Neural Language Modelscited by 163
  12. 2019Open Sesame: Getting Inside BERT's Linguistic Knowledgecited by 119
9 more
  1. 2018Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networkscited by 101
  2. 2020Does syntax need to grow on trees? Sources of hierarchical inductive bias in sequence-to-sequence networkscited by 72
  3. 2019Investigating BERT's Knowledge of Language: Five Analysis Methods with NPIscited by 58
  4. 2019Sequential Neural Networks as Automatacited by 54
  5. 2019Unsupervised Recurrent Neural Network Grammarscited by 47
  6. 2020Can neural networks acquire a structural bias from raw linguistic data?cited by 35
  7. 2019Quantity doesn't buy quality syntax with neural language modelscited by 25
  8. 2019Finding Syntactic Representations in Neural Stackscited by 4
  9. 2020Sequence-to-Sequence Networks Learn the Meaning of Reflexive Anaphoracited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 2; 2023: 8; 2024: 4; 2025: 3; 2026: 1.

19 papers.

Sort:

  1. 2022What Artificial Neural Networks Can Tell Us About Human Language Acquisitioncited by 97
  2. 2023Language Model Behavior: A Comprehensive Surveycited by 92
  3. 2023Grokking of Hierarchical Structure in Vanilla Transformerscited by 53
  4. 2022Coloring the Blank Slate: Pre-training Imparts a Hierarchical Inductive Bias to Sequence-to-sequence Modelscited by 28
  5. 2024Learning Syntax Without Planting Trees: Understanding Hierarchical Generalization in Transformerscited by 24
  6. 2023How to Plant Trees in Language Models: Data and Architectural Effects on the Emergence of Syntactic Inductive Biasescited by 23
  7. 2025Can Language Models Learn Typologically Implausible Languages?cited by 23
  8. 2023How poor is the stimulus? Evaluating hierarchical generalization in neural networks trained on child-directed speechcited by 22
  9. 2023In-context Learning Generalizes, But Not Always Robustly: The Case of Syntaxcited by 22
  10. 2023Stack Attention: Improving the Ability of Transformers to Model Hierarchical Patternscited by 17
  11. 2023How Abstract Is Linguistic Generalization in Large Language Models? Experiments with Argument Structurecited by 13
  12. 2024Sometimes I am a Tree: Data Drives Unstable Hierarchical Generalizationcited by 10
7 more
  1. 2021Causal Transformers Perform Below Chance on Recursive Nested Constructions, Unlike Humanscited by 8
  2. 2024Tree Transformers are an Ineffective Model of Syntactic Constituencycited by 3
  3. 2026Structure Before Collapse: Transient semantic geometry in next-token predictioncited by 2
  4. 2024Theoretical Analysis of Hierarchical Language Recognition and Generation by Transformers without Positional Encodingcited by 1
  5. 2025Bearing Syntactic Fruit with Stack-Augmented Neural Networkscited by 1
  6. 2023Nondeterministic Stacks in Neural Networkscited by 0
  7. 2025Exploring Compositional Generalization (in COGS/ReCOGS_pos) by Transformers using Restricted Access Sequence Processing (RASP)cited by 0