mlsanityData to 7 Oct 2026

Self-generated Replay Memories for Continual Neural Machine Translation

arXiv 2403.13130 ↗First posted 19 Mar 2024cs.CL cs.AI cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

Modern Neural Machine Translation systems exhibit strong performance in several different languages and are constantly improving. Their ability to learn continuously is, however, still severely limited by the catastrophic forgetting issue. In this work, we leverage a key property of encoder-decoder Transformers, i.e. their generative ability, to propose a novel approach to continually learning Neural Machine Translation systems. We show how this can effectively learn on a stream of experiences comprising different languages, by leveraging a replay memory populated by using the model itself as a generator of parallel sentences. We empirically demonstrate that our approach can counteract catastrophic forgetting without requiring explicit memorization of training data. Code will be publicly available upon publication. Code: https://github.com/m-resta/sg-rep

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 34 of its 49 references that are in the map. First posted on arXiv 19 Mar 2024.

Builds on

34 of 49 references.

References · 34

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  4. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  5. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  6. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  7. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  8. 2017Gradient Episodic Memory for Continual Learningcited by 1,871
  9. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  10. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  11. 2016Progressive Neural Networkscited by 1,332
  12. 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,094
22 more
  1. 2018Efficient Lifelong Learning with A-GEMcited by 1,044
  2. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  3. 2015Improving Neural Machine Translation Models with Monolingual Datacited by 784
  4. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
  5. 2019Three scenarios for continual learningcited by 610
  6. 2020Beyond English-Centric Multilingual Machine Translationcited by 559
  7. 2017Learning multiple visual domains with residual adapterscited by 553
  8. 2017PathNet: Evolution Channels Gradient Descent in Super Neural Networkscited by 362
  9. 2019MASS: Masked Sequence to Sequence Pre-training for Language Generationcited by 345
  10. 2019Massively Multilingual Neural Machine Translation in the Wild: Findings and Challengescited by 253
  11. 2015Net2Net: Accelerating Learning via Knowledge Transfercited by 245
  12. 2019LAMOL: LAnguage MOdeling for Lifelong Language Learningcited by 156
  13. 2021Avalanche: an End-to-End Library for Continual Learningcited by 124
  14. 2021LFPT5: A Unified Framework for Lifelong Few-shot Language Learning Based on Prompt Tuning of T5cited by 103
  15. 2019Jointly Measuring Diversity and Quality in Text Generation Modelscited by 61
  16. 2019Non-Parametric Adaptation for Neural Machine Translationcited by 34
  17. 2019Regularization Shortcomings for Continual Learningcited by 32
  18. 2022Continual Sequence Generation with Adaptive Compositional Modulescited by 31
  19. 2022Overcoming Catastrophic Forgetting beyond Continual Learning: Balanced Training for Neural Machine Translationcited by 18
  20. 2021Pruning-then-Expanding Model for Domain Adaptation of Neural Machine Translationcited by 15
  21. 2019From Bilingual to Multilingual Neural Machine Translation by Incremental Trainingcited by 14
  22. 2021Continual Learning in Multilingual NMT via Language-Specific Embeddingscited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 1; 2026: 2.

4 papers.

  1. 2024Towards Lifelong Learning of Large Language Models: A Surveycited by 60
  2. 2026Forgetting in Language Models: Capacity, Optimization, and Self-Generated Replaycited by 5
  3. 2025Neural Networks Remember More: The Power of Parameter Isolation and Combinationcited by 1
  4. 2026Continual Learning via Self-Probe Gradientscited by 0