mlsanityData to 7 Oct 2026

Coloring the Blank Slate: Pre-training Imparts a Hierarchical Inductive Bias to Sequence-to-sequence Models

arXiv 2203.09397 ↗First posted 17 Mar 2022cs.CLcited by 28

Not on the map. Why?

Abstract (arXiv)

Relations between words are governed by hierarchical structure rather than linear ordering. Sequence-to-sequence (seq2seq) models, despite their success in downstream NLP applications, often fail to generalize in a hierarchy-sensitive manner when performing syntactic transformations - for example, transforming declarative sentences into questions. However, syntactic evaluations of seq2seq models have only observed models that were not pre-trained on natural language data before being trained to perform syntactic transformations, in spite of the fact that pre-training has been found to induce hierarchical linguistic generalizations in language models; in other words, the syntactic capabilities of seq2seq models may have been greatly understated. We address this gap using the pre-trained seq2seq models T5 and BART, as well as their multilingual variants mT5 and mBART. We evaluate whether they generalize hierarchically on two transformations in two languages: question formation and passivization in English and German. We find that pre-trained seq2seq models generalize hierarchically when performing syntactic transformations, whereas models trained from scratch on syntactic transformations do not. This result presents evidence for the learnability of hierarchical syntactic information from non-annotated natural language text while also demonstrating that seq2seq models are capable of syntactic generalization, though only after exposure to much more language data than human learners receive.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Builds on 27 of its 42 references that are in the map. First posted on arXiv 17 Mar 2022.

Builds on

27 of 42 references.

References · 27

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  6. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  7. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  8. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  9. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  10. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 864
  11. 2016Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependenciescited by 350
  12. 2018Targeted Syntactic Evaluation of Language Modelscited by 239
15 more
  1. 2019Assessing BERT's Syntactic Abilitiescited by 191
  2. 2020A Systematic Assessment of Syntactic Generalization in Neural Language Modelscited by 163
  3. 2021Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Modelscited by 127
  4. 2020How Can We Accelerate Progress Towards Human-like Linguistic Generalization?cited by 114
  5. 2020Finding Universal Grammatical Relations in Multilingual BERTcited by 108
  6. 2018What do RNN Language Models Learn about Filler-Gap Dependencies?cited by 86
  7. 2020Does syntax need to grow on trees? Sources of hierarchical inductive bias in sequence-to-sequence networkscited by 72
  8. 2021Counterfactual Interventions Reveal the Causal Effect of Relative Clause Representations on Agreement Predictioncited by 58
  9. 2020Cross-Linguistic Syntactic Evaluation of Word Prediction Modelscited by 54
  10. 2021The Low-Dimensional Linear Geometry of Contextualized Word Representationscited by 39
  11. 2019Studying the Inductive Biases of RNNs with Synthetic Variations of Natural Languagescited by 37
  12. 2020Can neural networks acquire a structural bias from raw linguistic data?cited by 35
  13. 2018Revisiting the poverty of the stimulus: hierarchical generalization without a hierarchical bias in recurrent neural networkscited by 33
  14. 2019Quantity doesn't buy quality syntax with neural language modelscited by 25
  15. 2021Transformers Generalize Linearlycited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 12; 2024: 7; 2025: 3; 2026: 2.

28 papers.

Sort:

By area:

  1. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 97
  2. 2023A Theory of Emergent In-Context Learning as Implicit Structure Inductioncited by 86
  3. 2025Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corporacited by 86
  4. 2023Grokking of Hierarchical Structure in Vanilla Transformerscited by 53
  5. 2023Measuring Inductive Biases of In-Context Learning with Underspecified Demonstrationscited by 49
  6. 2023The Impact of Depth on Compositional Generalization in Transformer Language Modelscited by 37
  7. 2023Injecting structural hints: Using language models to study inductive biases in language learningcited by 33
  8. 2024How Does Code Pretraining Affect Language Model Task Performance?cited by 28
  9. 2024Learning Syntax Without Planting Trees: Understanding Hierarchical Generalization in Transformerscited by 24
  10. 2022Unit Testing for Concepts in Neural Networkscited by 23
  11. 2023How to Plant Trees in Language Models: Data and Architectural Effects on the Emergence of Syntactic Inductive Biasescited by 23
  12. 2022Language model acceptability judgements are not always robust to contextcited by 22
16 more
  1. 2023How poor is the stimulus? Evaluating hierarchical generalization in neural networks trained on child-directed speechcited by 22
  2. 2023In-context Learning Generalizes, But Not Always Robustly: The Case of Syntaxcited by 22
  3. 2023Stack Attention: Improving the Ability of Transformers to Model Hierarchical Patternscited by 17
  4. 2022Causal Analysis of Syntactic Agreement Neurons in Multilingual Language Modelscited by 14
  5. 2023How Abstract Is Linguistic Generalization in Large Language Models? Experiments with Argument Structurecited by 13
  6. 2024Sometimes I am a Tree: Data Drives Unstable Hierarchical Generalizationcited by 10
  7. 2025Subword models struggle with word learning, but surprisal hides itcited by 6
  8. 2024Strengthening Structural Inductive Biases by Pre-training to Perform Syntactic Transformationscited by 4
  9. 2024Experimental Contexts Can Facilitate Robust Semantic Property Inference in Language Models, but Inconsistentlycited by 3
  10. 2024Tree Transformers are an Ineffective Model of Syntactic Constituencycited by 3
  11. 2023Does Vision Accelerate Hierarchical Generalization in Neural Language Learners?cited by 1
  12. 2024Can Language Models Induce Grammatical Knowledge from Indirect Evidence?cited by 1
  13. 2025Bearing Syntactic Fruit with Stack-Augmented Neural Networkscited by 1
  14. 2023Exploiting Representation Bias for Data Distillation in Abstractive Text Summarizationcited by 0
  15. 2026Collocational bootstrapping: A hypothesis about the learning of subject-verb agreement in humans and neural networkscited by 0
  16. 2026Recurrent Reasoning on Symbolic Puzzles with Sequence Modelscited by 0