# Compositional Generalization for Primitive Substitutions

arXiv: https://arxiv.org/abs/1910.02612 · HTML: https://mlsanity.com/paper/1910.02612/

## Key facts

- Cited by 48 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Oct 2019.

## Built on by (most cited first, 30 of 48)

- [Least-to-Most Prompting Enables Complex Reasoning in Large Language Models](https://mlsanity.com/paper/2205.10625/) (2022)
- [The Devil is in the Detail: Simple Tricks Improve Systematic Generalization of Transformers](https://mlsanity.com/paper/2108.12284/) (2021)
- [State-of-the-art generalisation research in NLP: A taxonomy and review](https://mlsanity.com/paper/2210.03050/) (2022)
- [Compositional Semantic Parsing with Large Language Models](https://mlsanity.com/paper/2209.15003/) (2022)
- [Meta-Learning to Compositionally Generalize](https://mlsanity.com/paper/2106.04252/) (2021)
- [The Neural Data Router: Adaptive Control Flow in Transformers Improves Systematic Generalization](https://mlsanity.com/paper/2110.07732/) (2021)
- [Unlocking Compositional Generalization in Pre-trained Models Using Intermediate Representations](https://mlsanity.com/paper/2104.07478/) (2021)
- [Evaluating the Impact of Model Scale for Compositional Generalization in Semantic Parsing](https://mlsanity.com/paper/2205.12253/) (2022)
- [Improving Compositional Generalization with Latent Structure and Data Augmentation](https://mlsanity.com/paper/2112.07610/) (2021)
- [On Compositional Generalization of Neural Machine Translation](https://mlsanity.com/paper/2105.14802/) (2021)
- [Sparse Universal Transformer](https://mlsanity.com/paper/2310.07096/) (2023)
- [Learning Algebraic Recombination for Compositional Generalization](https://mlsanity.com/paper/2107.06516/) (2021)
- [Sequence-to-Sequence Learning with Latent Neural Grammars](https://mlsanity.com/paper/2109.01135/) (2021)
- [Finding needles in a haystack: Sampling Structurally-diverse Training Sets from Synthetic Data for Compositional Generalization](https://mlsanity.com/paper/2109.02575/) (2021)
- [On the Compositional Generalization Gap of In-Context Learning](https://mlsanity.com/paper/2211.08473/) (2022)
- [Position: Understanding LLMs Requires More Than Statistical Generalization](https://mlsanity.com/paper/2405.01964/) (2024)
- [Revisiting the Compositional Generalization Abilities of Neural Sequence Models](https://mlsanity.com/paper/2203.07402/) (2022)
- [Prototype-Guided Memory Replay for Continual Learning](https://mlsanity.com/paper/2108.12641/) (2021)
- [Inducing Transformer's Compositional Generalization Ability via Auxiliary Sequence Prediction Tasks](https://mlsanity.com/paper/2109.15256/) (2021)
- [Variational Cross-Graph Reasoning and Adaptive Structured Semantics Learning for Compositional Temporal Grounding](https://mlsanity.com/paper/2301.09071/) (2023)
- [ExeDec: Execution Decomposition for Compositional Generalization in Neural Program Synthesis](https://mlsanity.com/paper/2307.13883/) (2023)
- [Can Transformers Jump Around Right in Natural Language? Assessing Performance Transfer from SCAN](https://mlsanity.com/paper/2107.01366/) (2021)
- [Compositional Attention: Disentangling Search and Retrieval](https://mlsanity.com/paper/2110.09419/) (2021)
- [Neural-Symbolic Recursive Machine for Systematic Generalization](https://mlsanity.com/paper/2210.01603/) (2022)
- [When Can Transformers Ground and Compose: Insights from Compositional Generalization Benchmarks](https://mlsanity.com/paper/2210.12786/) (2022)
- [Induced Natural Language Rationales and Interleaved Markup Tokens Enable Extrapolation in Large Language Models](https://mlsanity.com/paper/2208.11445/) (2022)
- [Mutual Exclusivity Training and Primitive Augmentation to Induce Compositionality](https://mlsanity.com/paper/2211.15578/) (2022)
- [Learning to Substitute Spans towards Improving Compositional Generalization](https://mlsanity.com/paper/2306.02840/) (2023)
- [MAGNIFICo: Evaluating the In-Context Learning Ability of Large Language Models to Generalize to Novel Interpretations](https://mlsanity.com/paper/2310.11634/) (2023)
- [From Frege to chatGPT: Compositionality in language, cognition, and deep neural networks](https://mlsanity.com/paper/2405.15164/) (2024)

## Abstract (arXiv)

Compositional generalization is a basic mechanism in human language learning, but current neural networks lack such ability. In this paper, we conduct fundamental research for encoding compositionality in neural networks. Conventional methods use a single representation for the input sentence, making it hard to apply prior knowledge of compositionality. In contrast, our approach leverages such knowledge with two representations, one generating attention maps, and the other mapping attended input words to output symbols. We reduce the entropy in each representation to improve generalization. Our experiments demonstrate significant improvements over the conventional methods in five NLP tasks including instruction learning and machine translation. In the SCAN domain, it boosts accuracies from 14.0% to 98.8% in Jump task, and from 92.0% to 99.7% in TurnLeft task. It also beats human performance on a few-shot learning task. We hope the proposed approach can help ease future research towards human-level compositional language learning.