mlsanityData to 7 Oct 2026

DC-MBR: Distributional Cooling for Minimum Bayesian Risk Decoding

arXiv 2212.04205 ↗First posted 8 Dec 2022cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

Minimum Bayesian Risk Decoding (MBR) emerges as a promising decoding algorithm in Neural Machine Translation. However, MBR performs poorly with label smoothing, which is surprising as label smoothing provides decent improvement with beam search and improves generality in various tasks. In this work, we show that the issue arises from the un-consistency of label smoothing on the token-level and sequence-level distributions. We demonstrate that even though label smoothing only causes a slight change in the token-level, the sequence-level distribution is highly skewed. We coin the issue autoregressive over-smoothness. To address this issue, we propose a simple and effective method, Distributional Cooling MBR (DC-MBR), which manipulates the entropy of output distributions by tuning down the Softmax temperature. We theoretically prove the equivalence between pre-tuning label smoothing factor and distributional cooling. Extensive experiments on NMT benchmarks validate that distributional cooling improves MBR in various settings.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 62 of its 111 references that are in the map. First posted on arXiv 8 Dec 2022.

Builds on

62 of 111 references.

References · 62

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  4. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  5. 2017On Calibration of Modern Neural Networkscited by 4,860
  6. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  7. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  9. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  10. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  11. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  12. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
38 more
  1. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  2. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  3. 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,024
  4. 2020COMET: A Neural Framework for MT Evaluationcited by 1,024
  5. 2019When Does Label Smoothing Help?cited by 985
  6. 2017Convolutional Sequence to Sequence Learningcited by 550
  7. 2017Regularizing Neural Networks by Penalizing Confident Output Distributionscited by 461
  8. 2019MASS: Masked Sequence to Sequence Pre-training for Language Generationcited by 345
  9. 2017Six Challenges for Neural Machine Translationcited by 334
  10. 2019Learning Deep Transformer Models for Machine Translationcited by 309
  11. 2020Multilingual Translation with Extensible Multilingual Pretraining and Finetuningcited by 287
  12. 2017Beam Search Strategies for Neural Machine Translationcited by 232
  13. 2022In-context Examples Selection for Machine Translationcited by 172
  14. 2021To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translationcited by 161
  15. 2020Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metricscited by 155
  16. 2020Is MAP Decoding All You Need? The Inadequacy of the Mode in Neural Machine Translationcited by 137
  17. 2018Analyzing Uncertainty in Neural Machine Translationcited by 126
  18. 2019Bridging the Gap between Training and Inference for Neural Machine Translationcited by 123
  19. 2022Truncation Sampling as Language Model Desmoothingcited by 121
  20. 2018Correcting Length Bias in Neural Machine Translationcited by 119
  21. 2019On NMT Search Errors and Model Errors: Cat Got Your Tongue?cited by 114
  22. 2020If beam search is the answer, what was the question?cited by 114
  23. 2015Yara Parser: A Fast and Accurate Dependency Parsercited by 110
  24. 2021High Quality Rather than High Model Probability: Minimum Bayes Risk Decoding with Neural Metricscited by 104
  25. 2020Rethinking embedding coupling in pre-trained language modelscited by 103
  26. 2018The Best of Both Worlds: Combining Recent Advances in Neural Machine Translationcited by 101
  27. 2016Towards better decoding and language model integration in sequence to sequence modelscited by 100
  28. 2016Reward Augmented Maximum Likelihood for Neural Structured Predictioncited by 80
  29. 2018A Study of Reinforcement Learning for Neural Machine Translationcited by 74
  30. 2019Calibration of Encoder Decoder Models for Neural Machine Translationcited by 67
  31. 2021Sampling-Based Approximations to Minimum Bayes Risk Decoding for Neural Machine Translationcited by 63
  32. 2020On the Inference Calibration of Neural Machine Translationcited by 61
  33. 2021Understanding the Properties of Minimum Bayes Risk Decoding in Neural Machine Translationcited by 56
  34. 2021Machine Translation Decoding beyond Beam Searchcited by 55
  35. 2022Follow the Wisdom of the Crowd: Effective Text Generation via Minimum Bayes Risk Decodingcited by 54
  36. 2020Generalized Entropy Regularization or: There's Nothing Special about Label Smoothingcited by 40
  37. 2020Shallow-to-Deep Training for Neural Machine Translationcited by 36
  38. 2018Entity-Duet Neural Ranking: Understanding the Role of Knowledge Graph Semantics in Neural Information Retrievalcited by 33

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 2; 2025: 1.

4 papers.

  1. 2023It's MBR All the Way Down: Modern Generation Techniques Through the Lens of Minimum Bayes Riskcited by 49
  2. 2024Regression-aware Inference with LLMscited by 13
  3. 2024Machine Translation Models are Zero-Shot Detectors of Translation Directioncited by 3
  4. 2025Theoretical Guarantees for Minimum Bayes Risk Decodingcited by 3