# Rearranging the Familiar: Testing Compositional Generalization in Recurrent Networks

arXiv: https://arxiv.org/abs/1807.07545 · HTML: https://mlsanity.com/paper/1807.07545/

## Key facts

- Cited by 51 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Jul 2018.

## Built on by (most cited first, 30 of 51)

- [Case-based Reasoning for Natural Language Queries over Knowledge Bases](https://mlsanity.com/paper/2104.08762/) (2021)
- [State-of-the-art generalisation research in NLP: A taxonomy and review](https://mlsanity.com/paper/2210.03050/) (2022)
- [Compositional Semantic Parsing with Large Language Models](https://mlsanity.com/paper/2209.15003/) (2022)
- [Compositional Generalization from First Principles](https://mlsanity.com/paper/2307.05596/) (2023)
- [Unlocking Compositional Generalization in Pre-trained Models Using Intermediate Representations](https://mlsanity.com/paper/2104.07478/) (2021)
- [On Compositional Generalization of Neural Machine Translation](https://mlsanity.com/paper/2105.14802/) (2021)
- [Dissociating language and thought in large language models](https://mlsanity.com/paper/2301.06627/) (2023)
- [Neural-Logic Human-Object Interaction Detection](https://mlsanity.com/paper/2311.09817/) (2023)
- [Dynamic Inference with Neural Interpreters](https://mlsanity.com/paper/2110.06399/) (2021)
- [Equi-Tuning: Group Equivariant Fine-Tuning of Pretrained Models](https://mlsanity.com/paper/2210.06475/) (2022)
- [On Provable Length and Compositional Generalization](https://mlsanity.com/paper/2402.04875/) (2024)
- [(Un)solving Morphological Inflection: Lemma Overlap Artificially Inflates Models' Performance](https://mlsanity.com/paper/2108.05682/) (2021)
- [Inducing Transformer's Compositional Generalization Ability via Auxiliary Sequence Prediction Tasks](https://mlsanity.com/paper/2109.15256/) (2021)
- [Structured Reordering for Modeling Latent Alignments in Sequence Transduction](https://mlsanity.com/paper/2106.03257/) (2021)
- [SUBS: Subtree Substitution for Compositional Semantic Parsing](https://mlsanity.com/paper/2205.01538/) (2022)
- [How Modular Should Neural Module Networks Be for Systematic Generalization?](https://mlsanity.com/paper/2106.08170/) (2021)
- [Can Transformers Jump Around Right in Natural Language? Assessing Performance Transfer from SCAN](https://mlsanity.com/paper/2107.01366/) (2021)
- [Improving Systematic Generalization Through Modularity and Augmentation](https://mlsanity.com/paper/2202.10745/) (2022)
- [Sequential Query Encoding For Complex Query Answering on Knowledge Graphs](https://mlsanity.com/paper/2302.13114/) (2023)
- [The Role of Syntactic Planning in Compositional Image Captioning](https://mlsanity.com/paper/2101.11911/) (2021)
- [The Validity of Evaluation Results: Assessing Concurrence Across Compositionality Benchmarks](https://mlsanity.com/paper/2310.17514/) (2023)
- [Structurally Diverse Sampling for Sample-Efficient Training and Comprehensive Evaluation](https://mlsanity.com/paper/2203.08445/) (2022)
- [Defending Compositionality in Emergent Languages](https://mlsanity.com/paper/2206.04751/) (2022)
- [Mutual Exclusivity Training and Primitive Augmentation to Induce Compositionality](https://mlsanity.com/paper/2211.15578/) (2022)
- [Generalization in Multimodal Language Learning from Simulation](https://mlsanity.com/paper/2108.02319/) (2021)
- [Visual Referential Games Further the Emergence of Disentangled Representations](https://mlsanity.com/paper/2304.14511/) (2023)
- [Data Factors for Better Compositional Generalization](https://mlsanity.com/paper/2311.04420/) (2023)
- [Compositional Program Generation for Few-Shot Systematic Generalization](https://mlsanity.com/paper/2309.16467/) (2023)
- [Understanding Robust Generalization in Learning Regular Languages](https://mlsanity.com/paper/2202.09717/) (2022)
- [Learning Parameterized Task Structure for Generalization to Unseen Entities](https://mlsanity.com/paper/2203.15034/) (2022)

## Abstract (arXiv)

Systematic compositionality is the ability to recombine meaningful units with regular and predictable outcomes, and it's seen as key to humans' capacity for generalization in language. Recent work has studied systematic compositionality in modern seq2seq models using generalization to novel navigation instructions in a grounded environment as a probing tool, requiring models to quickly bootstrap the meaning of new words. We extend this framework here to settings where the model needs only to recombine well-trained functional words (such as "around" and "right") in novel contexts. Our findings confirm and strengthen the earlier ones: seq2seq models can be impressively good at generalizing to novel combinations of previously-seen input, but only when they receive extensive training on the specific pattern to be generalized (e.g., generalizing from many examples of "X around right" to "jump around right"), while failing when generalization requires novel application of compositional rules (e.g., inferring the meaning of "around right" from those of "right" and "around").