mlsanityData to 7 Oct 2026

Pruning-then-Expanding Model for Domain Adaptation of Neural Machine Translation

arXiv 2103.13678 ↗First posted 25 Mar 2021cs.CLcited by 15

Not on the map. Why?

Abstract (arXiv)

Domain Adaptation is widely used in practical applications of neural machine translation, which aims to achieve good performance on both the general-domain and in-domain. However, the existing methods for domain adaptation usually suffer from catastrophic forgetting, domain divergence, and model explosion. To address these three problems, we propose a method of "divide and conquer" which is based on the importance of neurons or parameters in the translation model. In our method, we first prune the model and only keep the important neurons or parameters, making them responsible for both general-domain and in-domain translation. Then we further train the pruned model supervised by the original unpruned model with the knowledge distillation method. Last we expand the model to the original size and fine-tune the added parameters for the in-domain translation. We conduct experiments on different languages and domains and the results show that our method can achieve significant improvements compared with several strong baselines.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 32 of its 50 references that are in the map. First posted on arXiv 25 Mar 2021.

Builds on

32 of 50 references.

References · 32

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  4. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  5. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  6. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  7. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  8. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  9. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  10. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  11. 2016Sequence-Level Knowledge Distillationcited by 761
  12. 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 596
20 more
  1. 2017Convolutional Sequence to Sequence Learningcited by 550
  2. 2019Simple, Scalable Adaptation for Neural Machine Translationcited by 239
  3. 2019Bridging the Gap between Training and Inference for Neural Machine Translationcited by 123
  4. 2018Identifying and Controlling Important Neurons in Neural Machine Translationcited by 110
  5. 2019Curriculum Learning for Domain Adaptation in Neural Machine Translationcited by 59
  6. 2020Efficient Meta Lifelong-Learning with Limited Memorycited by 48
  7. 2016Compression of Neural Machine Translation Models via Pruningcited by 43
  8. 2016Domain Control for Neural Machine Translationcited by 36
  9. 2016Fast Domain Adaptation for Neural Machine Translationcited by 28
  10. 2017Regularization techniques for fine-tuning in neural machine translationcited by 27
  11. 2018Extreme Adaptation for Personalized Neural Machine Translationcited by 27
  12. 2018Compact Personalized Models for Neural Machine Translationcited by 19
  13. 2020Investigating Catastrophic Forgetting During Continual Training for Neural Machine Translationcited by 17
  14. 2019Go From the General to the Particular: Multi-Domain Translation with Domain Transformation Networkscited by 12
  15. 2019Domain Adaptive Inference for Neural Machine Translationcited by 10
  16. 2019Iterative Dual Domain Adaptation for Neural Machine Translationcited by 10
  17. 2019Improving Domain Adaptation Translation with Domain Invariant and Specific Informationcited by 7
  18. 2020On the Sparsity of Neural Machine Translation Modelscited by 5
  19. 2017An Empirical Comparison of Simple Domain Adaptation Methods for Neural Machine Translationcited by 3
  20. 2020Finding Sparse Structures for Domain Specific Neural Machine Translationcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 4; 2023: 2; 2024: 2; 2025: 1; 2026: 1.

15 papers.

Sort:

  1. 2021Neuron-level Interpretation of Deep NLP Models: A Surveycited by 81
  2. 2022Discovering Latent Concepts Learned in BERTcited by 56
  3. 2021Domain Adaptation and Multi-Domain Adaptation for Neural Machine Translation: A Surveycited by 46
  4. 2021Importance-based Neuron Allocation for Multilingual Neural Machine Translationcited by 23
  5. 2022Overcoming Catastrophic Forgetting beyond Continual Learning: Balanced Training for Neural Machine Translationcited by 18
  6. 2023NeuroX Library for Neuron Analysis of Deep NLP Modelscited by 11
  7. 2022PADA: Pruning Assisted Domain Adaptation for Self-Supervised Speech Representationscited by 10
  8. 2021What do End-to-End Speech Models Learn about Speaker, Language and Channel Information? A Layer-wise and Neuron-level Analysiscited by 9
  9. 2022Continual Learning of Neural Machine Translation within Low Forgetting Risk Regionscited by 6
  10. 2021Fine-grained Interpretation and Causation Analysis in Deep NLP Modelscited by 4
  11. 2024Self-generated Replay Memories for Continual Neural Machine Translationcited by 4
  12. 2025IG-Pruning: Input-Guided Block Pruning for Large Language Modelscited by 2
3 more
  1. 2026KD4MT: A Survey of Knowledge Distillation for Machine Translationcited by 2
  2. 2024F-MALLOC: Feed-forward Memory Allocation for Continual Learning in Neural Machine Translationcited by 1
  3. 2023Exploiting Language Relatedness in Machine Translation Through Domain Adaptation Techniquescited by 0