mlsanityData to 14 Oct 2026

On Compositional Generalization of Neural Machine Translation

arXiv 2105.14802 ↗First posted 31 May 2021cs.CL cs.AI cs.LGcited by 34

Not on the map. Why?

Abstract (arXiv)

Modern neural machine translation (NMT) models have achieved competitive performance in standard benchmarks such as WMT. However, there still exist significant issues such as robustness, domain generalization, etc. In this paper, we study NMT models from the perspective of compositional generalization by building a benchmark dataset, CoGnition, consisting of 216k clean and consistent sentence pairs. We quantitatively analyze effects of various factors using compound translation error rate, then demonstrate that the NMT model fails badly on compositional generalization, although it performs remarkably well under traditional metrics.

Key facts. Cited by 34 papers in the map (data through 14 Oct 2026). Builds on 37 of its 57 references that are in the map. First posted on arXiv 31 May 2021.

Builds on

37 of 57 references.

References · 37

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  5. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  6. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  7. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,457
  8. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 856
  9. 2019Are Sixteen Heads Really Better than One?cited by 816
  10. 2020Stanza: A Python Natural Language Processing Toolkit for Many Human Languagescited by 723
  11. 2016A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Storiescited by 417
  12. 2019Analysing Mathematical Reasoning Abilities of Neural Modelscited by 335
25 more
  1. 2017Synthetic and Natural Noise Both Break Neural Machine Translationcited by 306
  2. 2019Measuring Compositional Generalization: A Comprehensive Method on Realistic Datacited by 284
  3. 2018Constituency Parsing with a Self-Attentive Encodercited by 199
  4. 2019Good-Enough Compositional Data Augmentationcited by 158
  5. 2019The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectivescited by 139
  6. 2018Achieving Human Parity on Automatic Chinese to English News Translationcited by 119
  7. 2019Robust Neural Machine Translation with Doubly Adversarial Inputscited by 106
  8. 2019Compositional generalization through meta sequence-to-sequence learningcited by 99
  9. 2018On Adversarial Examples for Character-Level Neural Machine Translationcited by 85
  10. 2021Lessons on Parameter Sharing across Layers in Transformerscited by 85
  11. 2020Compositional Generalization via Neural-Symbolic Stack Machinescited by 80
  12. 2020On the Inference Calibration of Neural Machine Translationcited by 61
  13. 2018Evaluating Layers of Representation in Neural Machine Translation on Part-of-Speech and Semantic Tagging Taskscited by 56
  14. 2019Compositional generalization in a deep seq2seq model by separating syntax and semanticscited by 56
  15. 2018MTNT: A Testbed for Machine Translation of Noisy Textcited by 54
  16. 2018Rearranging the Familiar: Testing Compositional Generalization in Recurrent Networkscited by 51
  17. 2020Very Deep Transformers for Neural Machine Translationcited by 50
  18. 2019Compositional Generalization for Primitive Substitutionscited by 48
  19. 2018Towards Robust Neural Machine Translationcited by 35
  20. 2018Jump to better conclusions: SCAN both left and rightcited by 29
  21. 2019CNNs found to jump around more skillfully than RNNs: Compositional generalization in seq2seq convolutional networkscited by 22
  22. 2019Towards Understanding Neural Machine Translation with Word Importancecited by 19
  23. 2019On Compositionality in Neural Machine Translationcited by 14
  24. 2019Machine Translation of Restaurant Reviews: New Corpus for Domain Adaptation and Robustnesscited by 10
  25. 2020The Unreasonable Volatility of Neural Machine Translation Modelscited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 3; 2022: 7; 2023: 11; 2024: 7; 2025: 5; 2026: 1.

34 papers.

Sort:

By area:

  1. 2024Benchmarking Complex Instruction-Following with Multiple Constraints Compositioncited by 152
  2. 2021How much do language models copy from their training data? Evaluating linguistic novelty in text generation using RAVENcited by 124
  3. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 98
  4. 2021The paradox of the compositionality of natural language: a neural machine translation case studycited by 59
  5. 2024Insight Over Sight: Exploring the Vision-Knowledge Conflicts in Multimodal LLMscited by 38
  6. 2021Disentangled Sequence to Sequence Learning for Compositional Generalizationcited by 30
  7. 2022Can Transformer be Too Compositional? Analysing Idiom Processing in Neural Machine Translationcited by 28
  8. 2024A Survey on Compositional Learning of AI Models: Theoretical and Experimental Practicescited by 15
  9. 2023A Survey on Compositional Generalization in Applicationscited by 12
  10. 2023The Validity of Evaluation Results: Assessing Concurrence Across Compositionality Benchmarkscited by 12
  11. 2025Lost in Literalism: How Supervised Training Shapes Translationese in LLMscited by 11
  12. 2023Semantic Parsing for Conversational Question Answering over Knowledge Graphscited by 10
22 more
  1. 2023Explicit Syntactic Guidance for Neural Text Generationcited by 8
  2. 2022FactMix: Using a Few Labeled In-domain Examples to Generalize to Cross-domain Named Entity Recognitioncited by 7
  3. 2022Hierarchical Phrase-based Sequence-to-Sequence Learningcited by 7
  4. 2023SIP: Injecting a Structural Inductive Bias into a Seq2Seq Model by Simulationcited by 7
  5. 2025Analyzing the Inner Workings of Transformers in Compositional Generalizationcited by 6
  6. 2025Answer, Refuse, or Guess? Investigating Risk-Aware Decision Making in Language Modelscited by 6
  7. 2023Target-Side Augmentation for Document-Level Machine Translationcited by 5
  8. 2022Forming Trees with Treeformerscited by 3
  9. 2022Real-World Compositional Generalization with Disentangled Sequence-to-Sequence Learningcited by 3
  10. 2023Learning to Compose Representations of Different Encoder Layers towards Improving Compositional Generalizationcited by 3
  11. 2025CARMA: Enhanced Compositionality in LLMs via Advanced Regularisation and Mutual Information Alignmentcited by 3
  12. 2022Categorizing Semantic Representations for Neural Machine Translationcited by 2
  13. 2023Layer-wise Representation Fusion for Compositional Generalizationcited by 2
  14. 2024Evaluating Structural Generalization in Neural Machine Translationcited by 2
  15. 2026Reinforcement Learning for Compositional Generalization with Outcome-Level Optimizationcited by 2
  16. 2023Out-of-Distribution Generalization in Text Classification: Past, Present, and Futurecited by 1
  17. 2023Joint Dropout: Improving Generalizability in Low-Resource Neural Machine Translation through Phrase Pair Variablescited by 1
  18. 2023On Using Distribution-Based Compositionality Assessment to Evaluate Compositional Generalisation in Machine Translationcited by 1
  19. 2024Inducing Systematicity in Transformers by Attending to Structurally Quantized Embeddingscited by 1
  20. 2024What Have We Achieved on Non-autoregressive Translation?cited by 1
  21. 2024LexMatcher: Dictionary-centric Data Collection for LLM-based Machine Translationcited by 1
  22. 2025Multi-Sourced Compositional Generalization in Visual Question Answeringcited by 0