mlsanityData to 7 Oct 2026

Reward Gaming in Conditional Text Generation

arXiv 2211.08714 ↗First posted 16 Nov 2022cs.CL cs.AI cs.LGcited by 29

Not on the map. Why?

Cited by 20 in LLM alignment / RLHF / preference optimisation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

To align conditional text generation model outputs with desired behaviors, there has been an increasing focus on training the model using reinforcement learning (RL) with reward functions learned from human annotations. Under this framework, we identify three common cases where high rewards are incorrectly assigned to undesirable patterns: noise-induced spurious correlation, naturally occurring spurious correlation, and covariate shift. We show that even though learned metrics achieve high performance on the distribution of the data used to train the reward function, the undesirable patterns may be amplified during RL training of the text generation model. While there has been discussion about reward gaming in the RL or safety community, in this discussion piece, we would like to highlight reward gaming in the natural language generation (NLG) community using concrete conditional text generation examples and discuss potential fixes and areas for future work.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Builds on 55 of its 87 references that are in the map. First posted on arXiv 16 Nov 2022.

Builds on

55 of 87 references.

References · 55

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  6. 2020Learning to summarize from human feedbackcited by 2,562
  7. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  8. 2016Concrete Problems in AI Safetycited by 1,582
  9. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  10. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  11. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  12. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 779
38 more
  1. 2015Sequence Level Training with Recurrent Neural Networkscited by 568
  2. 2019Neural Text Generation with Unlikelihood Trainingcited by 489
  3. 2021Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translationcited by 333
  4. 2018Reward learning from human preferences and demonstrations in Ataricited by 314
  5. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 312
  6. 2019Emergent Tool Use From Multi-Agent Autocurriculacited by 308
  7. 2021Recursively Summarizing Books with Human Feedbackcited by 287
  8. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  9. 2019Google Research Football: A Novel Reinforcement Learning Environmentcited by 257
  10. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 244
  11. 2020Information-Theoretic Probing with Minimum Description Lengthcited by 230
  12. 2017Deal or No Deal? End-to-End Learning for Negotiation Dialoguescited by 202
  13. 2022Online Decision Transformercited by 194
  14. 2022Defining and Characterizing Reward Hackingcited by 184
  15. 2021Alignment of Language Agentscited by 167
  16. 2022RL with KL penalties is better viewed as Bayesian inferencecited by 130
  17. 2021Teach Me to Explain: A Review of Datasets for Explainable Natural Language Processingcited by 126
  18. 2019Generalization in Generation: A closer look at Exposure Biascited by 123
  19. 2020A Distributional Approach to Controlled Text Generationcited by 117
  20. 2019On NMT Search Errors and Model Errors: Cat Got Your Tongue?cited by 114
  21. 2019Reward Tampering Problems and Solutions in Reinforcement Learning: A Causal Influence Diagram Perspectivecited by 107
  22. 2022Why Exposure Bias Matters: An Imitation Learning Perspective of Error Accumulation in Language Generationcited by 106
  23. 2017Long Text Generation via Adversarial Training with Leaked Informationcited by 100
  24. 2018The Surprising Creativity of Digital Evolution: A Collection of Anecdotes from the Evolutionary Computation and Artificial Life Research Communitiescited by 78
  25. 2015How (not) to Train your Generative Model: Scheduled Sampling, Likelihood, Adversary?cited by 76
  26. 2018Multi-Reward Reinforced Summarization with Saliency and Entailmentcited by 73
  27. 2020Text Generation by Learning from Demonstrationscited by 64
  28. 2020Consistency of a Recurrent Language Model With Respect to Incomplete Decodingcited by 54
  29. 2022Identifying Weaknesses in Machine Translation Metrics Through Minimum Bayes Risk Decoding: A Case Study for COMETcited by 41
  30. 2022The CRINGE Loss: Learning what language not to modelcited by 39
  31. 2021Combining Feature and Instance Attribution to Detect Artifactscited by 38
  32. 2016What to do about non-standard (or non-canonical) language in NLPcited by 35
  33. 2021Revisiting the Weaknesses of Reinforcement Learning for Neural Machine Translationcited by 32
  34. 2020Learning to Evaluate Translation Beyond English: BLEURT Submissions to the WMT Metrics 2020 Shared Taskcited by 25
  35. 2018Learning to Run challenge solutions: Adapting reinforcement learning methods for neuromusculoskeletal environmentscited by 19
  36. 2021Relating Neural Text Degeneration to Exposure Biascited by 19
  37. 2020Human-Paraphrased References Improve Neural Machine Translationcited by 15
  38. 2021AgreeSum: Agreement-Oriented Multi-Document Summarizationcited by 10

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 7; 2024: 15; 2025: 4; 2026: 3.

29 papers.

Sort:

By area:

  1. 2023A Long Way to Go: Investigating Length Correlations in RLHFcited by 285
  2. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hackingcited by 195
  3. 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Modelscited by 184
  4. 2024Towards Guaranteed Safe AI: A Framework for Ensuring Robust and Reliable AI Systemscited by 94
  5. 2024Human Alignment of Large Language Models through Online Preference Optimisationcited by 90
  6. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMscited by 89
  7. 2024Offline Regularised Reinforcement Learning for Large Language Models Alignmentcited by 48
  8. 2024Margin-aware Preference Optimization for Aligning Diffusion Models without Referencecited by 45
  9. 2023Goodhart's Law in Reinforcement Learningcited by 37
  10. 2024InfAlign: Inference-aware language model alignmentcited by 34
  11. 2023Preference-grounded Token-level Guidance for Language Model Fine-tuningcited by 31
  12. 2024Post-hoc Reward Calibration: A Case Study on Length Biascited by 29
17 more
  1. 2024On the Limited Generalization Capability of the Implicit Reward Model Induced by Direct Preference Optimizationcited by 26
  2. 2024Stratified Prediction-Powered Inference for Hybrid Language Model Evaluationcited by 24
  3. 2024The Accuracy Paradox in RLHF: When Better Reward Models Don't Yield Better Language Modelscited by 19
  4. 2023Leveraging Implicit Feedback from Deployment Data in Dialoguecited by 14
  5. 2025On the Robustness of Reward Models for Language Model Alignmentcited by 13
  6. 2024Can a Bayesian Oracle Prevent Harm from an Agent?cited by 12
  7. 2023Extrapolative Controlled Sequence Generation via Iterative Refinementcited by 7
  8. 2024Rethinking the Role of Proxy Rewards in Language Model Alignmentcited by 7
  9. 2025CHARM: Calibrating Reward Models With Chatbot Arena Scorescited by 7
  10. 2025Bias Fitting to Mitigate Length Bias of Reward Model in RLHFcited by 7
  11. 2024Partial Identifiability and Misspecification in Inverse Reinforcement Learningcited by 4
  12. 2025Toward Evaluative Thinking: Meta Policy Optimization with Evolving Reward Modelscited by 4
  13. 2026Reinforcement Learning via Value Gradient Flowcited by 4
  14. 2024Speech Recognition With LLMs Adapted to Disordered Speech Using Reinforcement Learningcited by 2
  15. 2024QE-EBM: Using Quality Estimators as Energy Loss for Machine Translationcited by 0
  16. 2026A Vocabulary for Multi-Agent Automated Research Systemscited by 0
  17. 2026SAGA: Score-Weighted Adaptive Generation Alignment for Low-Resource Nordic Language Modelscited by 0