mlsanityData to 7 Oct 2026

Reward Augmented Maximum Likelihood for Neural Structured Prediction

arXiv 1609.00150 ↗First posted 1 Sep 2016cs.LGcited by 80

Not on the map. Why?

Abstract (arXiv)

A key problem in structured output prediction is direct optimization of the task reward function that matters for test evaluation. This paper presents a simple and computationally efficient approach to incorporate task reward into a maximum likelihood framework. By establishing a link between the log-likelihood and expected reward objectives, we show that an optimal regularized expected reward is achieved when the conditional distribution of the outputs given the inputs is proportional to their exponentiated scaled rewards. Accordingly, we present a framework to smooth the predictive probability of the outputs using their corresponding rewards. We optimize the conditional log-probability of augmented outputs that are sampled proportionally to their exponentiated scaled rewards. Experiments on neural sequence to sequence models for speech recognition and machine translation show notable improvements over a maximum likelihood baseline by using reward augmented maximum likelihood (RAML), where the rewards are defined as the negative edit distance between the outputs and the ground truth labels.

Key facts. Cited by 80 papers in the map (data through 7 Oct 2026). First posted on arXiv 1 Sep 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2019: 2; 2020: 0; 2021: 23; 2022: 12; 2023: 17; 2024: 8; 2025: 7; 2026: 11.

80 papers.

Sort:

By area:

  1. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 325
  2. 2022Quark: Controllable Text Generation with Reinforced Unlearningcited by 236
  3. 2019A Divergence Minimization Perspective on Imitation Learning Methodscited by 188
  4. 2022BRIO: Bringing Order to Abstractive Summarizationcited by 161
  5. 2021SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarizationcited by 131
  6. 2023Aligning Language Models with Preferences through f-divergence Minimizationcited by 118
  7. 2021The Factual Inconsistency Problem in Abstractive Text Summarization: A Surveycited by 79
  8. 2021Machine Translation Decoding beyond Beam Searchcited by 55
  9. 2024Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignmentcited by 50
  10. 2021Greedification Operators for Policy Optimization: Investigating Forward and Reverse KL Divergencescited by 40
  11. 2022Transformers are Meta-Reinforcement Learnerscited by 39
  12. 2025Supervised Fine Tuning on Curated Data is Reinforcement Learning (and can be improved)cited by 37
38 more
  1. 2021Efficient (Soft) Q-Learning for Text Generation with Limited Good Datacited by 35
  2. 2023Preference-grounded Token-level Guidance for Language Model Fine-tuningcited by 31
  3. 2021Modeling the Second Player in Distributionally Robust Optimizationcited by 29
  4. 2023Tailoring Language Generation Models under Total Variation Distancecited by 27
  5. 2024Beyond Sparse Rewards: Enhancing Reinforcement Learning with Language Model Critique in Text Generationcited by 22
  6. 2022One Reference Is Not Enough: Diverse Distillation with Reference Selection for Non-Autoregressive Translationcited by 20
  7. 2021Sequence-Level Training for Non-Autoregressive Neural Machine Translationcited by 18
  8. 2023Leftover Lunch: Advantage-based Offline Reinforcement Learning for Language Modelscited by 17
  9. 2021Co-Adaptation of Algorithmic and Implementational Innovations in Inference-based Deep Reinforcement Learningcited by 15
  10. 2022Robust Probabilistic Time Series Forecastingcited by 15
  11. 2021Learning from Perturbations: Diverse and Informative Dialogue Generation with Inverse Adversarial Trainingcited by 13
  12. 2022Latent State Marginalization as a Low-cost Approach for Improving Explorationcited by 13
  13. 2025SCAR: Shapley Credit Assignment for More Efficient RLHFcited by 13
  14. 2023Semi-Offline Reinforcement Learning for Optimized Text Generationcited by 12
  15. 2021Smoothing and Shrinking the Sparse Seq2Seq Search Spacecited by 11
  16. 2023Reinforcement Learning for Generative AI: A Surveycited by 11
  17. 2024Investigating the Histogram Loss in Regressioncited by 11
  18. 2021Optimization Issues in KL-Constrained Approximate Policy Iterationcited by 10
  19. 2022CipherDAug: Ciphertext based Data Augmentation for Neural Machine Translationcited by 9
  20. 2021Mixed Cross Entropy Loss for Neural Machine Translationcited by 8
  21. 2021Toward a `Standard Model' of Machine Learningcited by 8
  22. 2021Rethinking Data Augmentation for Low-Resource Neural Machine Translation: A Multi-Task Learning Approachcited by 8
  23. 2022Generative Cooperative Networks for Natural Language Generationcited by 8
  24. 2023Towards Controllable Diffusion Models via Reward-Guided Explorationcited by 8
  25. 2023EMO: Earth Mover Distance Optimization for Auto-Regressive Language Modelingcited by 7
  26. 2023Assessing the efficacy of large language models in generating accurate teacher responsescited by 6
  27. 2023Target-Side Augmentation for Document-Level Machine Translationcited by 5
  28. 2024Revisiting ASR Error Correction with Specialized Modelscited by 5
  29. 2021Data Augmentation for Text Generation Without Any Augmented Datacited by 4
  30. 2021Learning Graphon Autoencoders for Generative Graph Modelingcited by 4
  31. 2021End-to-End Neuro-Symbolic Architecture for Image-to-Image Reasoning Taskscited by 4
  32. 2021Amortized Noisy Channel Neural Machine Translationcited by 4
  33. 2022DC-MBR: Distributional Cooling for Minimum Bayesian Risk Decodingcited by 4
  34. 2023Beyond Traditional Teaching: The Potential of Large Language Models and Chatbots in Graduate Engineering Educationcited by 4
  35. 2023On Synthetic Data for Back Translationcited by 4
  36. 2023Beyond MLE: Convex Learning for Text Generationcited by 4
  37. 2022Focus on the Target's Vocabulary: Masked Label Smoothing for Machine Translationcited by 3
  38. 2022INFINITY: A Simple Yet Effective Unsupervised Framework for Graph-Text Mutual Conversioncited by 3

50 most cited of 80 All 80 (JSON)