mlsanityData to 14 Oct 2026

Learning to Generalize Compositionally by Transferring Across Semantic Parsing Tasks

arXiv 2111.05013 ↗First posted 9 Nov 2021cs.CL cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

Neural network models often generalize poorly to mismatched domains or distributions. In NLP, this issue arises in particular when models are expected to generalize compositionally, that is, to novel combinations of familiar words and constructions. We investigate learning representations that facilitate transfer learning from one compositional task to another: the representation and the task-specific layers of the models are strategically trained differently on a pre-finetuning task such that they generalize well on mismatched splits that require compositionality. We apply this method to semantic parsing, using three very different datasets, COGS, GeoQuery and SCAN, used alternately as the pre-finetuning and target task. Our method significantly improves compositional generalization over baselines on the test set of the target task, which is held out during fine-tuning. Ablation studies characterize the utility of the major steps in the proposed algorithm and support our hypothesis.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 37 of its 51 references that are in the map. First posted on arXiv 9 Nov 2021.

Builds on

37 of 51 references.

References · 37

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
  5. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  6. 2019When Does Label Smoothing Help?cited by 988
  7. 2017An Overview of Multi-Task Learning in Deep Neural Networkscited by 803
  8. 2019Measuring Compositional Generalization: A Comprehensive Method on Realistic Datacited by 284
  9. 2019Compositionality decomposed: how do neural networks generalise?cited by 265
  10. 2020COGS: A Compositional Generalization Challenge Based on Semantic Interpretationcited by 263
  11. 2018Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Taskscited by 215
  12. 2021Muppet: Massive Multi-task Representations with Pre-Finetuningcited by 192
25 more
  1. 2019Good-Enough Compositional Data Augmentationcited by 158
  2. 2020Compositional Generalization and Natural Language Variation: Can a Semantic Parsing Approach Handle Both?cited by 147
  3. 2018Improving Text-to-SQL Evaluation Methodologycited by 135
  4. 2020Exploring and Predicting Transferability across NLP Taskscited by 128
  5. 2020Intermediate-Task Transfer Learning with Pretrained Models for Natural Language Understanding: When and Why Does It Work?cited by 127
  6. 2019Well-Read Students Learn Better: On the Importance of Pre-training Compact Modelscited by 120
  7. 2018Meta-Learning for Low-Resource Neural Machine Translationcited by 105
  8. 2019Compositional generalization through meta sequence-to-sequence learningcited by 99
  9. 2020Compositional Generalization in Semantic Parsing: Pre-training vs. Specialized Architecturescited by 95
  10. 2020Compositional Generalization via Neural-Symbolic Stack Machinescited by 80
  11. 2020Span-based Semantic Parsing for Compositional Generalizationcited by 77
  12. 2020Learning Compositional Rules via Neural Program Synthesiscited by 74
  13. 2020Low-Resource Domain Adaptation for Compositional Task-Oriented Semantic Parsingcited by 72
  14. 2020Learning to Recombine and Resample Data for Compositional Generalizationcited by 65
  15. 2016A Joint Many-Task Model: Growing a Neural Network for Multiple NLP Taskscited by 62
  16. 2021Meta-Learning to Compositionally Generalizecited by 62
  17. 2019Compositional generalization in a deep seq2seq model by separating syntax and semanticscited by 56
  18. 2021Unlocking Compositional Generalization in Pre-trained Models Using Intermediate Representationscited by 55
  19. 2020Compositional Generalization by Learning Analytical Expressionscited by 53
  20. 2019Compositional Generalization for Primitive Substitutionscited by 48
  21. 2020Improving Compositional Generalization in Semantic Parsingcited by 42
  22. 2020Compositional Generalization via Semantic Taggingcited by 32
  23. 2021Learning Algebraic Recombination for Compositional Generalizationcited by 31
  24. 2018Syntactic Scaffolds for Semantic Structurescited by 27
  25. 2021Finding needles in a haystack: Sampling Structurally-diverse Training Sets from Synthetic Data for Compositional Generalizationcited by 27

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 3; 2023: 2.

6 papers.

  1. 2023How Do In-Context Examples Affect Compositional Generalization?cited by 51
  2. 2022Evaluating the Impact of Model Scale for Compositional Generalization in Semantic Parsingcited by 50
  3. 2021Improving Compositional Generalization with Latent Structure and Data Augmentationcited by 46
  4. 2023Break It Down: Evidence for Structural Compositionality in Neural Networkscited by 44
  5. 2022Induced Natural Language Rationales and Interleaved Markup Tokens Enable Extrapolation in Large Language Modelscited by 10
  6. 2022A Short Survey of Systematic Generalizationcited by 0