mlsanityData to 7 Oct 2026

Structural generalization is hard for sequence-to-sequence models

arXiv 2210.13050 ↗First posted 24 Oct 2022cs.CLcited by 24

Not on the map. Why?

Abstract (arXiv)

Sequence-to-sequence (seq2seq) models have been successful across many NLP tasks, including ones that require predicting linguistic structure. However, recent work on compositional generalization has shown that seq2seq models achieve very low accuracy in generalizing to linguistic structures that were not seen in training. We present new evidence that this is a general limitation of seq2seq models that is present not just in semantic parsing, but also in syntactic parsing and in text-to-text tasks, and that this limitation can often be overcome by neurosymbolic models that have linguistic knowledge built in. We further report on some experiments that give initial answers on the reasons for these limitations.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Builds on 25 of its 38 references that are in the map. First posted on arXiv 24 Oct 2022.

Builds on

25 of 38 references.

References · 25

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  4. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  5. 2019What do you learn from context? Probing for sentence structure in contextualized word representationscited by 436
  6. 2016Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependenciescited by 350
  7. 2019Theoretical Limitations of Self-Attention in Neural Sequence Modelscited by 286
  8. 2019Measuring Compositional Generalization: A Comprehensive Method on Realistic Datacited by 280
  9. 2019Compositionality decomposed: how do neural networks generalise?cited by 264
  10. 2020COGS: A Compositional Generalization Challenge Based on Semantic Interpretationcited by 260
  11. 2018Constituency Parsing with a Self-Attentive Encodercited by 196
  12. 2018Dissecting Contextual Word Embeddings: Architecture and Representationcited by 146
13 more
  1. 2021The Devil is in the Detail: Simple Tricks Improve Systematic Generalization of Transformerscited by 117
  2. 2020Compositional Generalization in Semantic Parsing: Pre-training vs. Specialized Architecturescited by 95
  3. 2020Does syntax need to grow on trees? Sources of hierarchical inductive bias in sequence-to-sequence networkscited by 72
  4. 2021Making Transformers Solve Compositional Taskscited by 72
  5. 2021Meta-Learning to Compositionally Generalizecited by 62
  6. 2017A Minimal Span-Based Neural Constituency Parsercited by 59
  7. 2021Improving Compositional Generalization with Latent Structure and Data Augmentationcited by 46
  8. 2021Learning Algebraic Recombination for Compositional Generalizationcited by 31
  9. 2021Disentangled Sequence to Sequence Learning for Compositional Generalizationcited by 30
  10. 2022Unobserved Local Structures Make Compositional Generalization Hardcited by 29
  11. 2021Lexicon Learning for Few-Shot Neural Sequence Modelingcited by 28
  12. 2017Modeling Source Syntax for Neural Machine Translationcited by 12
  13. 2021Improving Compositional Generalization in Classification Tasks via Structure Annotationscited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 9; 2024: 6; 2025: 5; 2026: 4.

24 papers.

Sort:

By area:

  1. 2023ReCOGS: How Incidental Details of a Logical Form Overshadow an Evaluation of Semantic Interpretationcited by 24
  2. 2023What's the Meaning of Superhuman Performance in Today's NLU?cited by 22
  3. 2023SLOG: A Structural Generalization Benchmark for Semantic Parsingcited by 22
  4. 2025Anything Goes? A Crosslinguistic Study of (Im)possible Language Learning in LMscited by 20
  5. 2023The Validity of Evaluation Results: Assessing Concurrence Across Compositionality Benchmarkscited by 12
  6. 2023Leveraging Code to Improve In-context Learning for Semantic Parsingcited by 10
  7. 2025Analyzing the Inner Workings of Transformers in Compositional Generalizationcited by 6
  8. 2023Compositional Generalization without Trees using Multiset Tagging and Latent Permutationscited by 5
  9. 2024Strengthening Structural Inductive Biases by Pre-training to Perform Syntactic Transformationscited by 4
  10. 2023Learning to Compose Representations of Different Encoder Layers towards Improving Compositional Generalizationcited by 3
  11. 2023Layer-wise Representation Fusion for Compositional Generalizationcited by 2
  12. 2024Evaluating Structural Generalization in Neural Machine Translationcited by 2
12 more
  1. 2024Scope-enhanced Compositional Semantic Parsing for DRTcited by 2
  2. 2024Infusing Prompts with Syntax and Semanticscited by 2
  3. 2023Predicting generalization performance with correctness discriminatorscited by 1
  4. 2024Towards Compositionally Generalizable Semantic Parsing in Large Language Models: A Surveycited by 1
  5. 2024Out-of-distribution generalisation in spoken language understandingcited by 1
  6. 2025Randomly Sampled Language Reasoning Problems Elucidate Limitations of In-Context Learningcited by 1
  7. 2025ASPERA: A Simulated Environment to Evaluate Planning for Complex Action Executioncited by 1
  8. 2025DocPolarBERT: A Pre-trained Model for Document Understanding with Relative Polar Coordinate Encoding of Layout Structurescited by 0
  9. 2026Structural Generalization on SLOG without Hand-Written Rulescited by 0
  10. 2026Language Models Can Resolve Reference Compositionally, But It's Not Their Native Strength: The Case of the Personal Relation Taskcited by 0
  11. 2026On the Role of Directionality in Structural Generalizationcited by 0
  12. 2026On the Computational Complexity of Structural Generalizationcited by 0