mlsanityData to 14 Oct 2026

Memorisation Cartography: Mapping out the Memorisation-Generalisation Continuum in Neural Machine Translation

arXiv 2311.05379 ↗First posted 9 Nov 2023cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

When training a neural network, it will quickly memorise some source-target mappings from your dataset but never learn some others. Yet, memorisation is not easily expressed as a binary feature that is good or bad: individual datapoints lie on a memorisation-generalisation continuum. What determines a datapoint's position on that spectrum, and how does that spectrum influence neural models' performance? We address these two questions for neural machine translation (NMT) models. We use the counterfactual memorisation metric to (1) build a resource that places 5M NMT datapoints on a memorisation-generalisation map, (2) illustrate how the datapoints' surface-level characteristics and a models' per-datum training signals are predictive of memorisation in NMT, (3) and describe the influence that subsets of that map have on NMT systems' performance.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 28 of its 41 references that are in the map. First posted on arXiv 9 Nov 2023.

Builds on

28 of 41 references.

References · 28

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  3. 2020Extracting Training Data from Large Language Modelscited by 2,189
  4. 2016Understanding deep learning requires rethinking generalizationcited by 1,674
  5. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  6. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
  7. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 864
  8. 2018An Empirical Study of Example Forgetting during Deep Neural Network Learningcited by 678
  9. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 560
  10. 2019Benign Overfitting in Linear Regressioncited by 549
  11. 2020What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimationcited by 497
  12. 2019Does Learning Require Memorization? A Short Tale about a Long Tailcited by 460
16 more
  1. 2020Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamicscited by 425
  2. 2020Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translationcited by 277
  3. 2019WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipediacited by 223
  4. 2019CCMatrix: Mining Billions of High-Quality Parallel Sentences on the WEBcited by 177
  5. 2021The Curious Case of Hallucinations in Neural Machine Translationcited by 161
  6. 2021Counterfactual Memorization in Neural Language Modelscited by 156
  7. 2023Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4cited by 125
  8. 2022Understanding Transformer Memorization Recall Through Idiomscited by 62
  9. 2022Memorization in NLP Fine-tuning Methodscited by 45
  10. 2020How benign is benign overfitting?cited by 40
  11. 2021How BPE Affects Memorization in Transformerscited by 29
  12. 2021Language Modeling, Lexical Translation, Reordering: The Training Process of NMT through the Lens of Classical SMTcited by 21
  13. 2022An Empirical Study of Memorization in NLPcited by 21
  14. 2021A Tale Of Two Long Tailscited by 19
  15. 2022Measures of Information Reflect Memorization Patternscited by 12
  16. 2022Finding Memo: Extractive Memorization in Constrained Sequence Generation Taskscited by 12

Built on by

Cited by year

Citing papers per year: 2024: 3; 2025: 1.

4 papers.

  1. 2024Demystifying Verbatim Memorization in Large Language Modelscited by 56
  2. 2024Recite, Reconstruct, Recollect: Memorization in LMs as a Multifaceted Phenomenoncited by 31
  3. 2024Generalisation First, Memorisation Second? Memorisation Localisation for Natural Language Classification Taskscited by 9
  4. 2025Memorization Inheritance in Sequence-Level Knowledge Distillation for Neural Machine Translationcited by 2