mlsanityData to 7 Oct 2026

Mixed Cross Entropy Loss for Neural Machine Translation

arXiv 2106.15880 ↗First posted 30 Jun 2021cs.CLcited by 8

Not on the map. Why?

Abstract (arXiv)

In neural machine translation, cross entropy (CE) is the standard loss function in two training methods of auto-regressive models, i.e., teacher forcing and scheduled sampling. In this paper, we propose mixed cross entropy loss (mixed CE) as a substitute for CE in both training approaches. In teacher forcing, the model trained with CE regards the translation problem as a one-to-one mapping process, while in mixed CE this process can be relaxed to one-to-many. In scheduled sampling, we show that mixed CE has the potential to encourage the training and testing behaviours to be similar to each other, more effectively mitigating the exposure bias problem. We demonstrate the superiority of mixed CE over CE on several machine translation datasets, WMT'16 Ro-En, WMT'16 Ru-En, and WMT'14 En-De in both teacher forcing and scheduled sampling setups. Furthermore, in WMT'14 En-De, we also find mixed CE consistently outperforms CE on a multi-reference set as well as a challenging paraphrased reference set. We also found the model trained with mixed CE is able to provide a better probability distribution defined over the translation output space. Our code is available at https://github.com/haorannlp/mix.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 39 of its 54 references that are in the map. First posted on arXiv 30 Jun 2021.

Builds on

39 of 54 references.

References · 39

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  4. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  5. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  6. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  7. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  8. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  9. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  10. 2019When Does Label Smoothing Help?cited by 985
  11. 2015Improving Neural Machine Translation Models with Monolingual Datacited by 784
  12. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 779
27 more
  1. 2016Sequence-Level Knowledge Distillationcited by 761
  2. 2020Learning from Noisy Labels with Deep Neural Networks: A Surveycited by 580
  3. 2015Sequence Level Training with Recurrent Neural Networkscited by 568
  4. 2018Born Again Neural Networkscited by 539
  5. 2017Regularizing Neural Networks by Penalizing Confident Output Distributionscited by 461
  6. 2014Training Deep Neural Networks on Noisy Labels with Bootstrappingcited by 322
  7. 2017Synthetic and Natural Noise Both Break Neural Machine Translationcited by 304
  8. 2016Professor Forcing: A New Algorithm for Training Recurrent Networkscited by 214
  9. 2018Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinementcited by 194
  10. 2016An Actor-Critic Algorithm for Sequence Predictioncited by 158
  11. 2020On Exposure Bias, Hallucination and Domain Shift in Neural Machine Translationcited by 127
  12. 2016Sequence-to-Sequence Learning as Beam-Search Optimizationcited by 126
  13. 2018Analyzing Uncertainty in Neural Machine Translationcited by 126
  14. 2019Bridging the Gap between Training and Inference for Neural Machine Translationcited by 123
  15. 2019Generalization in Generation: A closer look at Exposure Biascited by 123
  16. 2015Minimum Risk Training for Neural Machine Translationcited by 114
  17. 2020BLEU might be Guilty but References are not Innocentcited by 109
  18. 2016Reward Augmented Maximum Likelihood for Neural Structured Predictioncited by 80
  19. 2019Mixture Models for Diverse Machine Translation: Tricks of the Tradecited by 77
  20. 2019Self-Knowledge Distillation in Natural Language Processingcited by 60
  21. 2017Classical Structured Prediction Losses for Sequence to Sequence Learningcited by 55
  22. 2019Scheduled Sampling for Transformerscited by 46
  23. 2018Denoising Neural Machine Translation Training with Trusted Data and Online Data Selectioncited by 43
  24. 2017SEARNN: Training RNNs with Global-Local Lossescited by 17
  25. 2019Parallel Scheduled Samplingcited by 17
  26. 2017Differentiable Scheduled Sampling for Credit Assignmentcited by 13
  27. 2019Controllable Dual Skew Divergence Loss for Neural Machine Translationcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 1; 2023: 2; 2024: 2; 2025: 1; 2026: 1.

8 papers.

  1. 2024GuardT2I: Defending Text-to-Image Models from Adversarial Promptscited by 61
  2. 2022Teacher Forcing Recovers Reward Functions for Text Generationcited by 11
  3. 2021Improving Scheduled Sampling with Elastic Weight Consolidation for Neural Machine Translationcited by 1
  4. 2023Magmaw: Modality-Agnostic Adversarial Attacks on Machine Learning-Based Wireless Communication Systemscited by 1
  5. 2023Dynamic Scheduled Sampling with Imitation Loss for Neural Text Generationcited by 0
  6. 2024Improving Autoregressive Training with Dynamic Oraclescited by 0
  7. 2025Relative-Based Scaling Law for Neural Language Modelscited by 0
  8. 2026LaCy: What Small Language Models Can and Should Learn is Not Just a Question of Losscited by 0