# Gradient-guided Loss Masking for Neural Machine Translation

arXiv: https://arxiv.org/abs/2102.13549 · HTML: https://mlsanity.com/paper/2102.13549/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 14 references that are in the map.
- First posted on arXiv 26 Feb 2021.

## Builds on (most cited first, 10 of 10)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [Learning to Reweight Examples for Robust Deep Learning](https://mlsanity.com/paper/1803.09050/) (2018)
- [On the Impact of Various Types of Noise on Neural Machine Translation](https://mlsanity.com/paper/1805.12282/) (2018)
- [Balancing Training for Multilingual Neural Machine Translation](https://mlsanity.com/paper/2004.06748/) (2020)
- [Optimizing Data Usage via Differentiable Rewards](https://mlsanity.com/paper/1911.10088/) (2019)
- [Denoising Neural Machine Translation Training with Trusted Data and Online Data Selection](https://mlsanity.com/paper/1809.00068/) (2018)
- [Dual Conditional Cross-Entropy Filtering of Noisy Parallel Corpora](https://mlsanity.com/paper/1809.00197/) (2018)
- [Dynamic Data Selection for Neural Machine Translation](https://mlsanity.com/paper/1708.00712/) (2017)

## Built on by (most cited first, 8 of 8)

- [Improving Multilingual Translation by Representation and Gradient Regularization](https://mlsanity.com/paper/2109.04778/) (2021)
- [On the Domain Adaptation and Generalization of Pretrained Language Models: A Survey](https://mlsanity.com/paper/2211.03154/) (2022)
- [The Trade-offs of Domain Adaptation for Neural Language Models](https://mlsanity.com/paper/2109.10274/) (2021)
- [Data Selection Curriculum for Neural Machine Translation](https://mlsanity.com/paper/2203.13867/) (2022)
- [On the Complementarity of Data Selection and Fine Tuning for Domain Adaptation](https://mlsanity.com/paper/2109.07591/) (2021)
- [Influence Functions for Sequence Tagging Models](https://mlsanity.com/paper/2210.14177/) (2022)
- [Por Qu'e N~ao Utiliser Alla Spraak? Mixed Training with Gradient Optimization in Few-Shot Cross-Lingual Transfer](https://mlsanity.com/paper/2204.13869/) (2022)
- [Switchable Representation Learning Framework with Self-compatibility](https://mlsanity.com/paper/2206.08289/) (2022)

## Abstract (arXiv)

To mitigate the negative effect of low quality training data on the performance of neural machine translation models, most existing strategies focus on filtering out harmful data before training starts. In this paper, we explore strategies that dynamically optimize data usage during the training process using the model's gradients on a small set of clean data. At each training step, our algorithm calculates the gradient alignment between the training data and the clean data to mask out data with negative alignment. Our method has a natural intuition: good training data should update the model parameters in a similar direction as the clean data. Experiments on three WMT language pairs show that our method brings significant improvement over strong baselines, and the improvements are generalizable across test data from different domains.