# Real-World Compositional Generalization with Disentangled Sequence-to-Sequence Learning

arXiv: https://arxiv.org/abs/2212.05982 · HTML: https://mlsanity.com/paper/2212.05982/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 49 references that are in the map.
- First posted on arXiv 12 Dec 2022.

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [Self-Attention with Relative Position Representations](https://mlsanity.com/paper/1803.02155/) (2018)
- [A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Stories](https://mlsanity.com/paper/1604.01696/) (2016)
- [Measuring Compositional Generalization: A Comprehensive Method on Realistic Data](https://mlsanity.com/paper/1912.09713/) (2019)
- [COGS: A Compositional Generalization Challenge Based on Semantic Interpretation](https://mlsanity.com/paper/2010.05465/) (2020)
- [Constrained Language Models Yield Few-Shot Semantic Parsers](https://mlsanity.com/paper/2104.08768/) (2021)
- [Good-Enough Compositional Data Augmentation](https://mlsanity.com/paper/1904.09545/) (2019)
- [Compositional Generalization and Natural Language Variation: Can a Semantic Parsing Approach Handle Both?](https://mlsanity.com/paper/2010.12725/) (2020)
- [Improving Text-to-SQL Evaluation Methodology](https://mlsanity.com/paper/1806.09029/) (2018)
- [Task-Oriented Dialogue as Dataflow Synthesis](https://mlsanity.com/paper/2009.11423/) (2020)
- [Data Recombination for Neural Semantic Parsing](https://mlsanity.com/paper/1606.03622/) (2016)
- [Coarse-to-Fine Decoding for Neural Semantic Parsing](https://mlsanity.com/paper/1805.04793/) (2018)
- [Span-based Semantic Parsing for Compositional Generalization](https://mlsanity.com/paper/2009.06040/) (2020)
- [Improve Transformer Models with Better Relative Position Embeddings](https://mlsanity.com/paper/2009.13658/) (2020)
- [Learning to Recombine and Resample Data for Compositional Generalization](https://mlsanity.com/paper/2010.03706/) (2020)
- [Meta-Learning to Compositionally Generalize](https://mlsanity.com/paper/2106.04252/) (2021)
- [The paradox of the compositionality of natural language: a neural machine translation case study](https://mlsanity.com/paper/2108.05885/) (2021)
- [Improving Compositional Generalization with Latent Structure and Data Augmentation](https://mlsanity.com/paper/2112.07610/) (2021)
- [Improving Compositional Generalization in Semantic Parsing](https://mlsanity.com/paper/2010.05647/) (2020)
- [Learning to Synthesize Data for Semantic Parsing](https://mlsanity.com/paper/2104.05827/) (2021)
- [On Compositional Generalization of Neural Machine Translation](https://mlsanity.com/paper/2105.14802/) (2021)
- [Compositional Generalization via Semantic Tagging](https://mlsanity.com/paper/2010.11818/) (2020)
- [Disentangled Sequence to Sequence Learning for Compositional Generalization](https://mlsanity.com/paper/2110.04655/) (2021)
- [Lexicon Learning for Few-Shot Neural Sequence Modeling](https://mlsanity.com/paper/2106.03993/) (2021)

## Built on by (most cited first, 3 of 3)

- [Learning to Compose Representations of Different Encoder Layers towards Improving Compositional Generalization](https://mlsanity.com/paper/2305.12169/) (2023)
- [Evaluating Structural Generalization in Neural Machine Translation](https://mlsanity.com/paper/2406.13363/) (2024)
- [On Using Distribution-Based Compositionality Assessment to Evaluate Compositional Generalisation in Machine Translation](https://mlsanity.com/paper/2311.08249/) (2023)

## Abstract (arXiv)

Compositional generalization is a basic mechanism in human language learning, which current neural networks struggle with. A recently proposed Disentangled sequence-to-sequence model (Dangle) shows promising generalization capability by learning specialized encodings for each decoding step. We introduce two key modifications to this model which encourage more disentangled representations and improve its compute and memory efficiency, allowing us to tackle compositional generalization in a more realistic setting. Specifically, instead of adaptively re-encoding source keys and values at each time step, we disentangle their representations and only re-encode keys periodically, at some interval. Our new architecture leads to better generalization performance across existing tasks and datasets, and a new machine translation benchmark which we create by detecting naturally occurring compositional patterns in relation to a training set. We show this methodology better emulates real-world requirements than artificial challenges.