# Disentangled Sequence to Sequence Learning for Compositional Generalization

arXiv: https://arxiv.org/abs/2110.04655 · HTML: https://mlsanity.com/paper/2110.04655/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 54 references that are in the map.
- First posted on arXiv 9 Oct 2021.

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Invariant Risk Minimization](https://mlsanity.com/paper/1907.02893/) (2019)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [Self-Attention with Relative Position Representations](https://mlsanity.com/paper/1803.02155/) (2018)
- [Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations](https://mlsanity.com/paper/1811.12359/) (2018)
- [Annotation Artifacts in Natural Language Inference Data](https://mlsanity.com/paper/1803.02324/) (2018)
- [Disentangling by Factorising](https://mlsanity.com/paper/1802.05983/) (2018)
- [A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Stories](https://mlsanity.com/paper/1604.01696/) (2016)
- [The Pitfalls of Simplicity Bias in Neural Networks](https://mlsanity.com/paper/2006.07710/) (2020)
- [Measuring Compositional Generalization: A Comprehensive Method on Realistic Data](https://mlsanity.com/paper/1912.09713/) (2019)
- [COGS: A Compositional Generalization Challenge Based on Semantic Interpretation](https://mlsanity.com/paper/2010.05465/) (2020)
- [Language to Logical Form with Neural Attention](https://mlsanity.com/paper/1601.01280/) (2016)
- [Good-Enough Compositional Data Augmentation](https://mlsanity.com/paper/1904.09545/) (2019)
- [Improving Text-to-SQL Evaluation Methodology](https://mlsanity.com/paper/1806.09029/) (2018)
- [Disentangled Representation Learning for Non-Parallel Text Style Transfer](https://mlsanity.com/paper/1808.04339/) (2018)
- [Data Recombination for Neural Semantic Parsing](https://mlsanity.com/paper/1606.03622/) (2016)
- [Compositional Generalization in Semantic Parsing: Pre-training vs. Specialized Architectures](https://mlsanity.com/paper/2007.08970/) (2020)
- [Span-based Semantic Parsing for Compositional Generalization](https://mlsanity.com/paper/2009.06040/) (2020)
- [Improve Transformer Models with Better Relative Position Embeddings](https://mlsanity.com/paper/2009.13658/) (2020)
- [Learning to Recombine and Resample Data for Compositional Generalization](https://mlsanity.com/paper/2010.03706/) (2020)
- [Meta-Learning to Compositionally Generalize](https://mlsanity.com/paper/2106.04252/) (2021)
- [Improving Disentangled Text Representation Learning with Information-Theoretic Guidance](https://mlsanity.com/paper/2006.00693/) (2020)
- [Improving Compositional Generalization in Semantic Parsing](https://mlsanity.com/paper/2010.05647/) (2020)
- [Towards Robust Neural Machine Translation](https://mlsanity.com/paper/1805.06130/) (2018)
- [On Compositional Generalization of Neural Machine Translation](https://mlsanity.com/paper/2105.14802/) (2021)
- [Compositional Generalization via Semantic Tagging](https://mlsanity.com/paper/2010.11818/) (2020)
- [Lexicon Learning for Few-Shot Neural Sequence Modeling](https://mlsanity.com/paper/2106.03993/) (2021)

## Built on by (most cited first, 30 of 30)

- [Benchmarking Complex Instruction-Following with Multiple Constraints Composition](https://mlsanity.com/paper/2407.03978/) (2024)
- [Does CLIP Bind Concepts? Probing Compositionality in Large Image Models](https://mlsanity.com/paper/2212.10537/) (2022)
- [State-of-the-art generalisation research in NLP: A taxonomy and review](https://mlsanity.com/paper/2210.03050/) (2022)
- [Sparse Universal Transformer](https://mlsanity.com/paper/2310.07096/) (2023)
- [Uncontrolled Lexical Exposure Leads to Overestimation of Compositional Generalization in Pretrained Models](https://mlsanity.com/paper/2212.10769/) (2022)
- [Structural generalization is hard for sequence-to-sequence models](https://mlsanity.com/paper/2210.13050/) (2022)
- [ReCOGS: How Incidental Details of a Logical Form Overshadow an Evaluation of Semantic Interpretation](https://mlsanity.com/paper/2303.13716/) (2023)
- [TAGPRIME: A Unified Framework for Relational Structure Extraction](https://mlsanity.com/paper/2205.12585/) (2022)
- [A Survey on Compositional Learning of AI Models: Theoretical and Experimental Practices](https://mlsanity.com/paper/2406.08787/) (2024)
- [Tripod: Three Complementary Inductive Biases for Disentangled Representation Learning](https://mlsanity.com/paper/2404.10282/) (2024)
- [Mutual Exclusivity Training and Primitive Augmentation to Induce Compositionality](https://mlsanity.com/paper/2211.15578/) (2022)
- [Learning to Substitute Spans towards Improving Compositional Generalization](https://mlsanity.com/paper/2306.02840/) (2023)
- [Seen to Unseen: Exploring Compositional Generalization of Multi-Attribute Controllable Dialogue Generation](https://mlsanity.com/paper/2306.10317/) (2023)
- [Compositional Generalization for Multi-label Text Classification: A Data-Augmentation Approach](https://mlsanity.com/paper/2312.11276/) (2023)
- [Compositional Generalisation with Structured Reordering and Fertility Layers](https://mlsanity.com/paper/2210.03183/) (2022)
- [Compositional Generalization without Trees using Multiset Tagging and Latent Permutations](https://mlsanity.com/paper/2305.16954/) (2023)
- [Blackbird's language matrices (BLMs): a new benchmark to investigate disentangled generalisation in neural networks](https://mlsanity.com/paper/2205.10866/) (2022)
- [SETI: Systematicity Evaluation of Textual Inference](https://mlsanity.com/paper/2305.15045/) (2023)
- [Recursive Decoding: A Situated Cognition Approach to Compositional Generation in Grounded Language Understanding](https://mlsanity.com/paper/2201.11766/) (2022)
- [Real-World Compositional Generalization with Disentangled Sequence-to-Sequence Learning](https://mlsanity.com/paper/2212.05982/) (2022)
- [Learning to Compose Representations of Different Encoder Layers towards Improving Compositional Generalization](https://mlsanity.com/paper/2305.12169/) (2023)
- [Compositional Generalization Requires Compositional Parsers](https://mlsanity.com/paper/2202.11937/) (2022)
- [Correcting Flaws in Common Disentanglement Metrics](https://mlsanity.com/paper/2304.02335/) (2023)
- [Exploring the Compositional Generalization in Context Dependent Text-to-SQL Parsing](https://mlsanity.com/paper/2306.04480/) (2023)
- [Layer-wise Representation Fusion for Compositional Generalization](https://mlsanity.com/paper/2307.10799/) (2023)
- [Out-of-Distribution Generalization in Text Classification: Past, Present, and Future](https://mlsanity.com/paper/2305.14104/) (2023)
- [Inducing Systematicity in Transformers by Attending to Structurally Quantized Embeddings](https://mlsanity.com/paper/2402.06492/) (2024)
- [Learning to Substitute Components for Compositional Generalization](https://mlsanity.com/paper/2502.20834/) (2025)
- [Disentangling continuous and discrete linguistic signals in transformer-based sentence embeddings](https://mlsanity.com/paper/2312.11272/) (2023)
- [Blackbird Language Matrices: A Framework to Investigate the Linguistic Competence of Language Models](https://mlsanity.com/paper/2602.20966/) (2026)

## Abstract (arXiv)

There is mounting evidence that existing neural network models, in particular the very popular sequence-to-sequence architecture, struggle to systematically generalize to unseen compositions of seen components. We demonstrate that one of the reasons hindering compositional generalization relates to representations being entangled. We propose an extension to sequence-to-sequence models which encourages disentanglement by adaptively re-encoding (at each time step) the source input. Specifically, we condition the source representations on the newly decoded target context which makes it easier for the encoder to exploit specialized information for each prediction rather than capturing it all in a single forward pass. Experimental results on semantic parsing and machine translation empirically show that our proposal delivers more disentangled representations and better generalization.