mlsanityData to 7 Oct 2026

Reward-Augmented Decoding: Efficient Controlled Text Generation With a Unidirectional Reward Model

arXiv 2310.09520 ↗First posted 14 Oct 2023cs.CLcited by 79

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

While large language models have proven effective in a huge range of downstream applications, they often generate text that is problematic or lacks a desired attribute. In this paper, we introduce Reward-Augmented Decoding (RAD), a text generation procedure that uses a small unidirectional reward model to encourage a language model to generate text that has certain properties. Specifically, RAD uses the reward model to score generations as they are produced and rescales sampling probabilities to favor high-reward tokens. By using a unidirectional reward model, RAD can cache activations from prior generation steps to decrease computational overhead. Through experiments on generating non-toxic and sentiment-controlled text, we demonstrate that RAD performs best among methods that change only the generation procedure and matches the performance of state-of-the-art methods that involve re-training the language model. We further validate that RAD is effective on very large language models while incurring a minimal computational overhead.

Key facts. Cited by 79 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 30 of its 34 references that are in the map. First posted on arXiv 14 Oct 2023.

Builds on

30 of 34 references.

References · 30

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2020Scaling Laws for Neural Language Modelscited by 6,411
  5. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  6. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  7. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  8. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  9. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  10. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  11. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  12. 2018Hierarchical Neural Story Generationcited by 1,182
18 more
  1. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,068
  2. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 754
  3. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 709
  4. 2021DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Expertscited by 434
  5. 2020GeDi: Generative Discriminator Guided Sequence Generationcited by 377
  6. 2021FUDGE: Controlled Text Generation With Future Discriminatorscited by 374
  7. 2018Generating Wikipedia by Summarizing Long Sequencescited by 333
  8. 2022Quark: Controllable Text Generation with Reinforced Unlearningcited by 236
  9. 2018Delete, Retrieve, Generate: A Simple Approach to Sentiment and Style Transfercited by 191
  10. 2019What makes a good conversation? How controllable attributes affect human judgmentscited by 140
  11. 2018Learning to Write with Cooperative Discriminatorscited by 119
  12. 2019Transforming Delete, Retrieve, Generate Approach for Controlled Text Style Transfercited by 81
  13. 2021A Plug-and-Play Method for Controlled Text Generationcited by 55
  14. 2022Critic-Guided Decoding for Controlled Text Generationcited by 45
  15. 2023On the Challenges of Using Black-Box APIs for Toxicity Evaluation in Researchcited by 44
  16. 2023Systematic Rectification of Language Models via Dead-end Analysiscited by 16
  17. 2022Classifiers are Better Experts for Controllable Text Generationcited by 10
  18. 2022Which Discriminator for Cooperative Text Generation?cited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 24; 2025: 39; 2026: 14.

79 papers.

Sort:

By area:

  1. 2025TTRL: Test-Time Reinforcement Learningcited by 253
  2. 2024From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Modelscited by 136
  3. 2024Weak-to-Strong Jailbreaking on Large Language Modelscited by 100
  4. 2024ARGS: Alignment as Reward-Guided Searchcited by 99
  5. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMscited by 89
  6. 2024Tuning Language Models by Proxycited by 88
  7. 2024Probabilistic Inference in Language Models via Twisted Sequential Monte Carlocited by 85
  8. 2025Inference-Time Alignment in Diffusion Models with Reward-Guided Generation: Tutorial and Reviewcited by 82
  9. 2024Decoding-time Realignment of Language Modelscited by 67
  10. 2023An Emulator for Fine-Tuning Large Language Models using Small Language Modelscited by 61
  11. 2024TreeBoN: Enhancing Inference-Time Alignment with Speculative Tree-Search and Best-of-N Samplingcited by 61
  12. 2024DeAL: Decoding-time Alignment for Large Language Modelscited by 55
38 more
  1. 2024Merge, Ensemble, and Cooperate! A Survey on Collaborative Strategies in the Era of Large Language Modelscited by 43
  2. 2023The Alignment Ceiling: Objective Mismatch in Reinforcement Learning from Human Feedbackcited by 42
  3. 2025Video-T1: Test-Time Scaling for Video Generationcited by 39
  4. 2024Cascade Reward Sampling for Efficient Decoding-Time Alignmentcited by 38
  5. 2024Controllable Safety Alignment: Inference-Time Adaptation to Diverse Safety Requirementscited by 35
  6. 2025SynthesizeMe! Inducing Persona-Guided Prompts for Personalized Reward Models in LLMscited by 26
  7. 2025A Survey on Personalized and Pluralistic Preference Alignment in Large Language Modelscited by 25
  8. 2024Nudging: Inference-time Alignment of LLMs via Guided Decodingcited by 22
  9. 2024Towards a Unified View of Preference Learning for Large Language Models: A Surveycited by 17
  10. 2024Selective Prompt Anchoring for Code Generationcited by 16
  11. 2024Diffusion Guided Language Modelingcited by 15
  12. 2025Control the Temperature: Selective Sampling for Diverse and High-Quality LLM Outputscited by 15
  13. 2025Sailing by the Stars: A Survey on Reward Models and Learning Strategies for Learning from Rewardscited by 14
  14. 2025Bone Soups: A Seek-and-Soup Model Merging Approach for Controllable Multi-Objective Generationcited by 13
  15. 2024Learning to Plan for Language Modeling from Unlabeled Datacited by 12
  16. 2024AdvisorQA: Towards Helpful and Harmless Advice-seeking Question Answering with Collective Intelligencecited by 12
  17. 2024A Critical Look At Tokenwise Reward-Guided Text Generationcited by 12
  18. 2025Personalized LLM Decoding via Contrasting Personal Preferencecited by 12
  19. 2025SPINE: Token-Selective Test-Time Reinforcement Learning with Entropy-Band Regularizationcited by 11
  20. 2025From Outcomes to Processes: Guiding PRM Learning from ORM for Inference-Time Alignmentcited by 9
  21. 2025Asking a Language Model for Diverse Responsescited by 9
  22. 2025Detoxifying Large Language Models via Autoregressive Reward Guided Representation Editingcited by 9
  23. 2024Testing the Limits of Jailbreaking Defenses with the Purple Problemcited by 8
  24. 2024Generalists vs. Specialists: Evaluating LLMs on Highly-Constrained Biophysical Sequence Optimization Taskscited by 8
  25. 2025Decoupling Task-Solving and Output Formatting in LLM Generationcited by 8
  26. 2024Large Language Models can be Strong Self-Detoxifierscited by 7
  27. 2025Inference-time Alignment in Continuous Spacecited by 7
  28. 2025IF-GUIDE: Influence Function-Guided Detoxification of LLMscited by 7
  29. 2025Inverse Reinforcement Learning Meets Large Language Model Post-Training: Basics, Advances, and Opportunitiescited by 7
  30. 2025Speculative Safety-Aware Decodingcited by 7
  31. 2024Constrained Decoding with Speculative Lookaheadscited by 6
  32. 2025Towards Cost-Effective Reward Guided Text Generationcited by 5
  33. 2025Simultaneous Multi-objective Alignment Across Verifiable and Non-verifiable Rewardscited by 5
  34. 2026ModeX: Evaluator-Free Best-of-N Selection for Open-Ended Generationcited by 5
  35. 2025Inference-Time Personalized Alignment with a Few User Preference Queriescited by 4
  36. 2024On the Low-Rank Parametrization of Reward Models for Controlled Language Generationcited by 3
  37. 2025Vulnerability Mitigation for Safety-Aligned Language Models via Debiasingcited by 3
  38. 2025Large-Scale Constraint Generation -- Can LLMs Parse Hundreds of Constraints?cited by 3

50 most cited of 79 All 79 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Beyond Sparse Rewards: Enhancing Reinforcement Learning with Language Model Critique in Text Generation
  2. 2024Improving Open-Ended Text Generation via Adaptive Decoding
  3. 2025reWordBench: Benchmarking and Improving the Robustness of Reward Models with Transformed Inputs
  4. 2024Reward-RAG: Enhancing RAG with Reward Driven Supervision
  5. 2018Toward Diverse Text Generation with Inverse Reinforcement Learning
  6. 2026Robust Reward Modeling for Large Language Models via Causal Decomposition
  7. 2026PARM: Pipeline-Adapted Reward Model
  8. 2025Retrieval-augmented Decoding for Improving Truthfulness in Open-ended Generation