# Mutual Information and Diverse Decoding Improve Neural Machine Translation

arXiv: https://arxiv.org/abs/1601.00372 · HTML: https://mlsanity.com/paper/1601.00372/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Jan 2016.

## Built on by (most cited first, 30 of 35)

- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [On Decoding Strategies for Neural Text Generators](https://mlsanity.com/paper/2203.15721/) (2022)
- [Jointly Reinforcing Diversity and Quality in Language Model Generations](https://mlsanity.com/paper/2509.02534/) (2025)
- [Corex: Pushing the Boundaries of Complex Reasoning through Multi-Model Collaboration](https://mlsanity.com/paper/2310.00280/) (2023)
- [Sampling-Based Approximations to Minimum Bayes Risk Decoding for Neural Machine Translation](https://mlsanity.com/paper/2108.04718/) (2021)
- [Revisiting the Role of Language Priors in Vision-Language Models](https://mlsanity.com/paper/2306.01879/) (2023)
- [Domain Adaptation and Multi-Domain Adaptation for Neural Machine Translation: A Survey](https://mlsanity.com/paper/2104.06951/) (2021)
- [Fast Nearest Neighbor Machine Translation](https://mlsanity.com/paper/2105.14528/) (2021)
- [The Consensus Game: Language Model Generation via Equilibrium Search](https://mlsanity.com/paper/2310.09139/) (2023)
- [Benchmarking and Improving Text-to-SQL Generation under Ambiguity](https://mlsanity.com/paper/2310.13659/) (2023)
- [Few-shot Subgoal Planning with Language Models](https://mlsanity.com/paper/2205.14288/) (2022)
- [Controllable Text Summarization: Unraveling Challenges, Approaches, and Prospects -- A Survey](https://mlsanity.com/paper/2311.09212/) (2023)
- [Metasql: A Generate-then-Rank Framework for Natural Language to SQL Translation](https://mlsanity.com/paper/2402.17144/) (2024)
- [ETS: Efficient Tree Search for Inference-Time Scaling](https://mlsanity.com/paper/2502.13575/) (2025)
- [Improving Group Robustness on Spurious Correlation Requires Preciser Group Inference](https://mlsanity.com/paper/2404.13815/) (2024)
- [ConRPG: Paraphrase Generation using Contexts as Regularizer](https://mlsanity.com/paper/2109.00363/) (2021)
- [Generating Diverse and High-Quality Texts by Minimum Bayes Risk Decoding](https://mlsanity.com/paper/2401.05054/) (2024)
- [Incentivizing Truthful Language Models via Peer Elicitation Games](https://mlsanity.com/paper/2505.13636/) (2025)
- [Faster Nearest Neighbor Machine Translation](https://mlsanity.com/paper/2112.08152/) (2021)
- [Pragmatic Reasoning improves LLM Code Generation](https://mlsanity.com/paper/2502.15835/) (2025)
- [Cooperative Self-training of Machine Reading Comprehension](https://mlsanity.com/paper/2103.07449/) (2021)
- [REAMsharp: An Enhancement Approach to Reference-based Evaluation Metrics for Open-domain Dialog Generation](https://mlsanity.com/paper/2105.14488/) (2021)
- [RewardsOfSum: Exploring Reinforcement Learning Rewards for Summarisation](https://mlsanity.com/paper/2106.04080/) (2021)
- [Truth or Deceit? A Bayesian Decoding Game Enhances Consistency and Reliability](https://mlsanity.com/paper/2410.01064/) (2024)
- [Geometry of Knowledge Allows Extending Diversity Boundaries of Large Language Models](https://mlsanity.com/paper/2507.13874/) (2025)
- [Multi-Amateur Contrastive Decoding for Text Generation](https://mlsanity.com/paper/2507.21086/) (2025)
- [Generating Diverse Translation with Perturbed kNN-MT](https://mlsanity.com/paper/2402.09344/) (2024)
- [Self-Explained Keywords Empower Large Language Models for Code Generation](https://mlsanity.com/paper/2410.15966/) (2024)
- [There Once Was a Really Bad Poet, It Was Automated but You Didn't Know It](https://mlsanity.com/paper/2103.03775/) (2021)
- [Goal-driven text descriptions for images](https://mlsanity.com/paper/2108.12575/) (2021)

## Abstract (arXiv)

Sequence-to-sequence neural translation models learn semantic and syntactic relations between sentence pairs by optimizing the likelihood of the target given the source, i.e., p(y|x), an objective that ignores other potentially useful sources of information. We introduce an alternative objective function for neural MT that maximizes the mutual information between the source and target sentences, modeling the bi-directional dependency of sources and targets. We implement the model with a simple re-ranking method, and also introduce a decoding algorithm that increases diversity in the N-best list produced by the first pass. Applied to the WMT German/English and French/English tasks, the proposed models offers a consistent performance boost on both standard LSTM and attention-based neural MT architectures.