# Reward Augmented Maximum Likelihood for Neural Structured Prediction

arXiv: https://arxiv.org/abs/1609.00150 · HTML: https://mlsanity.com/paper/1609.00150/

## Key facts

- Cited by 81 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Sep 2016.

## Built on by (most cited first, 30 of 81)

- [Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models](https://mlsanity.com/paper/2312.06585/) (2023)
- [Quark: Controllable Text Generation with Reinforced Unlearning](https://mlsanity.com/paper/2205.13636/) (2022)
- [A Divergence Minimization Perspective on Imitation Learning Methods](https://mlsanity.com/paper/1911.02256/) (2019)
- [BRIO: Bringing Order to Abstractive Summarization](https://mlsanity.com/paper/2203.16804/) (2022)
- [SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarization](https://mlsanity.com/paper/2106.01890/) (2021)
- [Aligning Language Models with Preferences through f-divergence Minimization](https://mlsanity.com/paper/2302.08215/) (2023)
- [The Factual Inconsistency Problem in Abstractive Text Summarization: A Survey](https://mlsanity.com/paper/2104.14839/) (2021)
- [Machine Translation Decoding beyond Beam Search](https://mlsanity.com/paper/2104.05336/) (2021)
- [Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment](https://mlsanity.com/paper/2412.14516/) (2024)
- [Greedification Operators for Policy Optimization: Investigating Forward and Reverse KL Divergences](https://mlsanity.com/paper/2107.08285/) (2021)
- [Transformers are Meta-Reinforcement Learners](https://mlsanity.com/paper/2206.06614/) (2022)
- [Supervised Fine Tuning on Curated Data is Reinforcement Learning (and can be improved)](https://mlsanity.com/paper/2507.12856/) (2025)
- [Efficient (Soft) Q-Learning for Text Generation with Limited Good Data](https://mlsanity.com/paper/2106.07704/) (2021)
- [Preference-grounded Token-level Guidance for Language Model Fine-tuning](https://mlsanity.com/paper/2306.00398/) (2023)
- [Modeling the Second Player in Distributionally Robust Optimization](https://mlsanity.com/paper/2103.10282/) (2021)
- [Tailoring Language Generation Models under Total Variation Distance](https://mlsanity.com/paper/2302.13344/) (2023)
- [Beyond Sparse Rewards: Enhancing Reinforcement Learning with Language Model Critique in Text Generation](https://mlsanity.com/paper/2401.07382/) (2024)
- [One Reference Is Not Enough: Diverse Distillation with Reference Selection for Non-Autoregressive Translation](https://mlsanity.com/paper/2205.14333/) (2022)
- [Sequence-Level Training for Non-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/2106.08122/) (2021)
- [Leftover Lunch: Advantage-based Offline Reinforcement Learning for Language Models](https://mlsanity.com/paper/2305.14718/) (2023)
- [SCAR: Shapley Credit Assignment for More Efficient RLHF](https://mlsanity.com/paper/2505.20417/) (2025)
- [Co-Adaptation of Algorithmic and Implementational Innovations in Inference-based Deep Reinforcement Learning](https://mlsanity.com/paper/2103.17258/) (2021)
- [Robust Probabilistic Time Series Forecasting](https://mlsanity.com/paper/2202.11910/) (2022)
- [Learning from Perturbations: Diverse and Informative Dialogue Generation with Inverse Adversarial Training](https://mlsanity.com/paper/2105.15171/) (2021)
- [Latent State Marginalization as a Low-cost Approach for Improving Exploration](https://mlsanity.com/paper/2210.00999/) (2022)
- [Semi-Offline Reinforcement Learning for Optimized Text Generation](https://mlsanity.com/paper/2306.09712/) (2023)
- [Smoothing and Shrinking the Sparse Seq2Seq Search Space](https://mlsanity.com/paper/2103.10291/) (2021)
- [Reinforcement Learning for Generative AI: A Survey](https://mlsanity.com/paper/2308.14328/) (2023)
- [Investigating the Histogram Loss in Regression](https://mlsanity.com/paper/2402.13425/) (2024)
- [Optimization Issues in KL-Constrained Approximate Policy Iteration](https://mlsanity.com/paper/2102.06234/) (2021)

## Abstract (arXiv)

A key problem in structured output prediction is direct optimization of the task reward function that matters for test evaluation. This paper presents a simple and computationally efficient approach to incorporate task reward into a maximum likelihood framework. By establishing a link between the log-likelihood and expected reward objectives, we show that an optimal regularized expected reward is achieved when the conditional distribution of the outputs given the inputs is proportional to their exponentiated scaled rewards. Accordingly, we present a framework to smooth the predictive probability of the outputs using their corresponding rewards. We optimize the conditional log-probability of augmented outputs that are sampled proportionally to their exponentiated scaled rewards. Experiments on neural sequence to sequence models for speech recognition and machine translation show notable improvements over a maximum likelihood baseline by using reward augmented maximum likelihood (RAML), where the rewards are defined as the negative edit distance between the outputs and the ground truth labels.