# Memorization Inheritance in Sequence-Level Knowledge Distillation for Neural Machine Translation

arXiv: https://arxiv.org/abs/2502.01491 · HTML: https://mlsanity.com/paper/2502.01491/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation.
- Builds on 30 of its 48 references that are in the map.
- First posted on arXiv 3 Feb 2025.

## Placed in

- Knowledge distillation / [Knowledge distillation for LLMs](https://mlsanity.com/distill/knowledge-distillation-for-llms/)

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [Sequence-Level Knowledge Distillation](https://mlsanity.com/paper/1606.07947/) (2016)
- [Dissecting Recall of Factual Associations in Auto-Regressive Language Models](https://mlsanity.com/paper/2304.14767/) (2023)
- [What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimation](https://mlsanity.com/paper/2008.03703/) (2020)
- [Does Learning Require Memorization? A Short Tale about a Long Tail](https://mlsanity.com/paper/1906.05271/) (2019)
- [Marian: Fast Neural Machine Translation in C++](https://mlsanity.com/paper/1804.00344/) (2018)
- [Sources of Hallucination by Large Language Models on Inference Tasks](https://mlsanity.com/paper/2305.14552/) (2023)
- [The Curious Case of Hallucinations in Neural Machine Translation](https://mlsanity.com/paper/2104.06683/) (2021)
- [Understanding Knowledge Distillation in Non-autoregressive Machine Translation](https://mlsanity.com/paper/1911.02727/) (2019)
- [f-Divergence Minimization for Sequence-Level Knowledge Distillation](https://mlsanity.com/paper/2307.15190/) (2023)
- [Building Machine Translation Systems for the Next Thousand Languages](https://mlsanity.com/paper/2205.03983/) (2022)
- [Understanding Transformer Memorization Recall Through Idioms](https://mlsanity.com/paper/2210.03588/) (2022)
- [Students Parrot Their Teachers: Membership Inference on Model Distillation](https://mlsanity.com/paper/2303.03446/) (2023)
- [Lifelong Language Knowledge Distillation](https://mlsanity.com/paper/2010.02123/) (2020)
- [Recite, Reconstruct, Recollect: Memorization in LMs as a Multifaceted Phenomenon](https://mlsanity.com/paper/2406.17746/) (2024)
- [SALTED: A Framework for SAlient Long-Tail Translation Error Detection](https://mlsanity.com/paper/2205.09988/) (2022)
- [Explaining Sequence-Level Knowledge Distillation as Data-Augmentation for Neural Machine Translation](https://mlsanity.com/paper/1912.03334/) (2019)
- [Finding Memo: Extractive Memorization in Constrained Sequence Generation Tasks](https://mlsanity.com/paper/2210.12929/) (2022)
- [Don't Throw Away Data: Better Sequence Knowledge Distillation](https://mlsanity.com/paper/2407.10456/) (2024)
- [ROME: Memorization Insights from Text, Logits and Representation](https://mlsanity.com/paper/2403.00510/) (2024)
- [What do larger image classifiers memorise?](https://mlsanity.com/paper/2310.05337/) (2023)
- [Life-long Learning for Multilingual Neural Machine Translation with Knowledge Distillation](https://mlsanity.com/paper/2212.02800/) (2022)
- [An Empirical Study of Leveraging Knowledge Distillation for Compressing Multilingual Neural Machine Translation Models](https://mlsanity.com/paper/2304.09388/) (2023)
- [Multi-Stage Balanced Distillation: Addressing Long-Tail Challenges in Sequence-Level Knowledge Distillation](https://mlsanity.com/paper/2406.13114/) (2024)
- [Memorisation Cartography: Mapping out the Memorisation-Generalisation Continuum in Neural Machine Translation](https://mlsanity.com/paper/2311.05379/) (2023)
- [ALBERTI, a Multilingual Domain Specific Language Model for Poetry Analysis](https://mlsanity.com/paper/2307.01387/) (2023)

## Built on by (most cited first, 2 of 2)

- [Memorization Dynamics in Knowledge Distillation for Language Models](https://mlsanity.com/paper/2601.15394/) (2026)
- [KD4MT: A Survey of Knowledge Distillation for Machine Translation](https://mlsanity.com/paper/2602.15845/) (2026)

## Abstract (arXiv)

In this work, we explore how instance-level memorization in the teacher Neural Machine Translation (NMT) model gets inherited by the student model in sequence-level knowledge distillation (SeqKD). We find that despite not directly seeing the original training data, students memorize more than baseline models (models of the same size, trained on the original data) -- 3.4% for exact matches and 57% for extractive memorization -- and show increased hallucination rates. Further, under this SeqKD setting, we also characterize how students behave on specific training data subgroups, such as subgroups with low quality and specific counterfactual memorization (CM) scores, and find that students exhibit amplified denoising on low-quality subgroups. Finally, we propose a modification to SeqKD named Adaptive-SeqKD, which intervenes in SeqKD to reduce memorization and hallucinations. Overall, we recommend caution when applying SeqKD: students inherit both their teachers' superior performance and their fault modes, thereby requiring active monitoring.