# Compositional generalization in a deep seq2seq model by separating syntax and semantics

arXiv: https://arxiv.org/abs/1904.09708 · HTML: https://mlsanity.com/paper/1904.09708/

## Key facts

- Cited by 56 papers in the map (data through 7 Oct 2026).
- First posted on arXiv 22 Apr 2019.

## Built on by (most cited first, 30 of 56)

- [Least-to-Most Prompting Enables Complex Reasoning in Large Language Models](https://mlsanity.com/paper/2205.10625/) (2022)
- [The Devil is in the Detail: Simple Tricks Improve Systematic Generalization of Transformers](https://mlsanity.com/paper/2108.12284/) (2021)
- [Compositional Semantic Parsing with Large Language Models](https://mlsanity.com/paper/2209.15003/) (2022)
- [Making Transformers Solve Compositional Tasks](https://mlsanity.com/paper/2108.04378/) (2021)
- [Meta-Learning to Compositionally Generalize](https://mlsanity.com/paper/2106.04252/) (2021)
- [The Neural Data Router: Adaptive Control Flow in Transformers Improves Systematic Generalization](https://mlsanity.com/paper/2110.07732/) (2021)
- [Compositional Generalization from First Principles](https://mlsanity.com/paper/2307.05596/) (2023)
- [Compositional Temporal Grounding with Structured Variational Cross-Graph Correspondence Learning](https://mlsanity.com/paper/2203.13049/) (2022)
- [Unlocking Compositional Generalization in Pre-trained Models Using Intermediate Representations](https://mlsanity.com/paper/2104.07478/) (2021)
- [Evaluating the Impact of Model Scale for Compositional Generalization in Semantic Parsing](https://mlsanity.com/paper/2205.12253/) (2022)
- [Improving Compositional Generalization with Latent Structure and Data Augmentation](https://mlsanity.com/paper/2112.07610/) (2021)
- [ShadowGNN: Graph Projection Neural Network for Text-to-SQL Parser](https://mlsanity.com/paper/2104.04689/) (2021)
- [On Compositional Generalization of Neural Machine Translation](https://mlsanity.com/paper/2105.14802/) (2021)
- [Learning Algebraic Recombination for Compositional Generalization](https://mlsanity.com/paper/2107.06516/) (2021)
- [Sequence-to-Sequence Learning with Latent Neural Grammars](https://mlsanity.com/paper/2109.01135/) (2021)
- [Lexicon Learning for Few-Shot Neural Sequence Modeling](https://mlsanity.com/paper/2106.03993/) (2021)
- [Sparse Universal Transformer](https://mlsanity.com/paper/2310.07096/) (2023)
- [The Relational Bottleneck as an Inductive Bias for Efficient Abstraction](https://mlsanity.com/paper/2309.06629/) (2023)
- [On the Compositional Generalization Gap of In-Context Learning](https://mlsanity.com/paper/2211.08473/) (2022)
- [Equi-Tuning: Group Equivariant Fine-Tuning of Pretrained Models](https://mlsanity.com/paper/2210.06475/) (2022)
- [Revisiting the Compositional Generalization Abilities of Neural Sequence Models](https://mlsanity.com/paper/2203.07402/) (2022)
- [Structured Reordering for Modeling Latent Alignments in Sequence Transduction](https://mlsanity.com/paper/2106.03257/) (2021)
- [ExeDec: Execution Decomposition for Compositional Generalization in Neural Program Synthesis](https://mlsanity.com/paper/2307.13883/) (2023)
- [Can Transformers Jump Around Right in Natural Language? Assessing Performance Transfer from SCAN](https://mlsanity.com/paper/2107.01366/) (2021)
- [Improving Systematic Generalization Through Modularity and Augmentation](https://mlsanity.com/paper/2202.10745/) (2022)
- [Information-Restricted Neural Language Models Reveal Different Brain Regions' Sensitivity to Semantics, Syntax and Context](https://mlsanity.com/paper/2302.14389/) (2023)
- [Improving Compositional Generalization in Classification Tasks via Structure Annotations](https://mlsanity.com/paper/2106.10434/) (2021)
- [Neural-Symbolic Recursive Machine for Systematic Generalization](https://mlsanity.com/paper/2210.01603/) (2022)
- [Induced Natural Language Rationales and Interleaved Markup Tokens Enable Extrapolation in Large Language Models](https://mlsanity.com/paper/2208.11445/) (2022)
- [Compositional Generalization for Multi-label Text Classification: A Data-Augmentation Approach](https://mlsanity.com/paper/2312.11276/) (2023)

## Abstract (arXiv)

Standard methods in deep learning for natural language processing fail to capture the compositional structure of human language that allows for systematic generalization outside of the training distribution. However, human learners readily generalize in this way, e.g. by applying known grammatical rules to novel words. Inspired by work in neuroscience suggesting separate brain systems for syntactic and semantic processing, we implement a modification to standard approaches in neural machine translation, imposing an analogous separation. The novel model, which we call Syntactic Attention, substantially outperforms standard methods in deep learning on the SCAN dataset, a compositional generalization task, without any hand-engineered features or additional supervision. Our work suggests that separating syntactic from semantic learning may be a useful heuristic for capturing compositional structure.