# On Compositionality in Neural Machine Translation

arXiv: https://arxiv.org/abs/1911.01497 · HTML: https://mlsanity.com/paper/1911.01497/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Nov 2019.

## Built on by (most cited first, 14 of 14)

- [State-of-the-art generalisation research in NLP: A taxonomy and review](https://mlsanity.com/paper/2210.03050/) (2022)
- [The paradox of the compositionality of natural language: a neural machine translation case study](https://mlsanity.com/paper/2108.05885/) (2021)
- [On Compositional Generalization of Neural Machine Translation](https://mlsanity.com/paper/2105.14802/) (2021)
- [How BPE Affects Memorization in Transformers](https://mlsanity.com/paper/2110.02782/) (2021)
- [Can Transformer be Too Compositional? Analysing Idiom Processing in Neural Machine Translation](https://mlsanity.com/paper/2205.15301/) (2022)
- [Symbolic Brittleness in Sequence Models: on Systematic Generalization in Symbolic Mathematics](https://mlsanity.com/paper/2109.13986/) (2021)
- [Searchable Hidden Intermediates for End-to-End Models of Decomposable Sequence Tasks](https://mlsanity.com/paper/2105.00573/) (2021)
- [Finding Memo: Extractive Memorization in Constrained Sequence Generation Tasks](https://mlsanity.com/paper/2210.12929/) (2022)
- [The Validity of Evaluation Results: Assessing Concurrence Across Compositionality Benchmarks](https://mlsanity.com/paper/2310.17514/) (2023)
- [Learning to Compose Representations of Different Encoder Layers towards Improving Compositional Generalization](https://mlsanity.com/paper/2305.12169/) (2023)
- [Categorizing Semantic Representations for Neural Machine Translation](https://mlsanity.com/paper/2210.06709/) (2022)
- [Layer-wise Representation Fusion for Compositional Generalization](https://mlsanity.com/paper/2307.10799/) (2023)
- [Evaluating Structural Generalization in Neural Machine Translation](https://mlsanity.com/paper/2406.13363/) (2024)
- [On Using Distribution-Based Compositionality Assessment to Evaluate Compositional Generalisation in Machine Translation](https://mlsanity.com/paper/2311.08249/) (2023)

## Abstract (arXiv)

We investigate two specific manifestations of compositionality in Neural Machine Translation (NMT) : (1) Productivity - the ability of the model to extend its predictions beyond the observed length in training data and (2) Systematicity - the ability of the model to systematically recombine known parts and rules. We evaluate a standard Sequence to Sequence model on tests designed to assess these two properties in NMT. We quantitatively demonstrate that inadequate temporal processing, in the form of poor encoder representations is a bottleneck for both Productivity and Systematicity. We propose a simple pre-training mechanism which alleviates model performance on the two properties and leads to a significant improvement in BLEU scores.