mlsanityData to 14 Oct 2026

Beyond Sparse Rewards: Enhancing Reinforcement Learning with Language Model Critique in Text Generation

arXiv 2401.07382 ↗First posted 14 Jan 2024cs.CL cs.AIcited by 22

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reinforcement learning (RL) can align language models with non-differentiable reward signals, such as human preferences. However, a major challenge arises from the sparsity of these reward signals - typically, there is only a single reward for an entire output. This sparsity of rewards can lead to inefficient and unstable learning. To address this challenge, our paper introduces an novel framework that utilizes the critique capability of Large Language Models (LLMs) to produce intermediate-step rewards during RL training. Our method involves coupling a policy model with a critic language model, which is responsible for providing comprehensive feedback of each part of the output. This feedback is then translated into token or span-level rewards that can be used to guide the RL training process. We investigate this approach under two different settings: one where the policy model is smaller and is paired with a more powerful critic model, and another where a single language model fulfills both roles. We assess our approach on three text generation tasks: sentiment control, language model detoxification, and summarization. Experimental results show that incorporating artificial intrinsic rewards significantly improve both sample efficiency and the overall performance of the policy model, supported by both automatic and human evaluation.

Key facts. Cited by 22 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 47 of its 65 references that are in the map. First posted on arXiv 14 Jan 2024.

Builds on

47 of 65 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback1 Sep
  2. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Training2 Jun
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  4. 2021WebGPT: Browser-assisted question-answering with human feedback17 Dec
  5. 2019Fine-Tuning Language Models from Human Preferences18 Sep

Other references · 42

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2023Let's Verify Step by Stepcited by 3,789
  5. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  6. 2020Learning to summarize from human feedbackcited by 2,588
  7. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  8. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  9. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  10. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  11. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,255
  12. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
30 more
  1. 2017Hindsight Experience Replaycited by 1,136
  2. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 799
  3. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 762
  4. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 639
  5. 2015Sequence Level Training with Recurrent Neural Networkscited by 576
  6. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 455
  7. 2021DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Expertscited by 441
  8. 2022Self-critiquing models for assisting human evaluatorscited by 380
  9. 2020GeDi: Generative Discriminator Guided Sequence Generationcited by 378
  10. 2016#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learningcited by 309
  11. 2016Deep Reinforcement Learning for Dialogue Generationcited by 288
  12. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 284
  13. 2017Count-Based Exploration with Neural Density Modelscited by 257
  14. 2022Quark: Controllable Text Generation with Reinforced Unlearningcited by 237
  15. 2016An Actor-Critic Algorithm for Sequence Predictioncited by 159
  16. 2016Dual Learning for Machine Translationcited by 148
  17. 2017Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Playcited by 141
  18. 2021Hallucinated but Factual! Inspecting the Factuality of Hallucinations in Abstractive Summarizationcited by 129
  19. 2019Using Natural Language for Reward Shaping in Reinforcement Learningcited by 114
  20. 2018On Learning Intrinsic Rewards for Policy Gradient Methodscited by 102
  21. 2023Motif: Intrinsic Motivation from Artificial Intelligence Feedbackcited by 102
  22. 2016Reward Augmented Maximum Likelihood for Neural Structured Predictioncited by 81
  23. 2023RewriteLM: An Instruction-Tuned Large Language Model for Text Rewritingcited by 67
  24. 2020Text Generation by Learning from Demonstrationscited by 64
  25. 2021Hierarchical Reinforcement Learning By Discovering Intrinsic Optionscited by 58
  26. 2017Ask the Right Questions: Active Question Reformulation with Reinforcement Learningcited by 44
  27. 2016Towards End-to-End Reinforcement Learning of Dialogue Agents for Information Accesscited by 36
  28. 2023Systematic Rectification of Language Models via Dead-end Analysiscited by 16
  29. 2016Policy Networks with Two-Stage Training for Dialogue Systemscited by 14
  30. 2017DCN+: Mixed Objective and Deep Residual Coattention for Question Answeringcited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 11; 2026: 9.

22 papers.

Sort:

By area:

  1. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 135
  2. 2025Ignore the KL Penalty! Boosting Exploration on Critical Tokens to Enhance RL Fine-Tuningcited by 34
  3. 2025Navigate the Unknown: Enhancing LLM Reasoning with Intrinsic Motivation Guided Explorationcited by 34
  4. 2025RewardMap: Tackling Sparse Rewards in Fine-grained Visual Reasoning via Multi-Stage Reinforcement Learningcited by 21
  5. 2025Text2Grad: Reinforcement Learning from Natural Language Feedbackcited by 18
  6. 2025A Survey on Progress in LLM Alignment from the Perspective of Reward Designcited by 15
  7. 2024A Critical Look At Tokenwise Reward-Guided Text Generationcited by 12
  8. 2025MuSC: Improving Complex Instruction Following with Multi-granularity Self-Contrastive Trainingcited by 12
  9. 2026Rubrics to Tokens: Bridging Response-level Rubrics and Token-level Rewards in Instruction Following Taskscited by 10
  10. 2024Extracting Heuristics from Large Language Models for Reward Shaping in Reinforcement Learningcited by 8
  11. 2026V0: A Generalist Value Model for Any Policy at State Zerocited by 7
  12. 2025Response-Level Rewards Are All You Need for Online Reinforcement Learning in LLMs: A Mathematical Perspectivecited by 5
10 more
  1. 2025Surfer 2: The Next Generation of Cross-Platform Computer Use Agentscited by 5
  2. 2026Towards Cold-Start Drafting and Continual Refining: A Value-Driven Memory Approach with Application to NPU Kernel Synthesiscited by 3
  3. 2025DPRM: A Dual Implicit Process Reward Model in Multi-Hop Question Answeringcited by 2
  4. 2026Modularized Reinforcement Learning on LLMs: From MDP Creation to Exploration and Learningcited by 2
  5. 2026Reward Models Are Secretly Value Functions: Temporally Coherent Reward Modelingcited by 1
  6. 2026PriorZero: Bridging Language Priors and World Models for Decision Makingcited by 1
  7. 2025Accelerating Reinforcement Learning Algorithms Convergence using Pre-trained Large Language Models as Tutors With Advice Reusingcited by 0
  8. 2026Best-of-Q: Improving VLM agents with Q-function Action Ranking at Inferencecited by 0
  9. 2026When Can You Poison Rewards? A Tight Characterization of Reward Poisoning in Linear MDPscited by 0
  10. 2026CATPO: Critique-Augmented Tree Policy Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Self-Generated Critiques Boost Reward Modeling for Language Models
  2. 2024Improving Reward Models with Synthetic Critiques
  3. 2025Towards Cost-Effective Reward Guided Text Generation
  4. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization
  5. 2026Reward Modeling from Natural Language Human Feedback
  6. 2022Critic-Guided Decoding for Controlled Text Generation
  7. 2024West-of-N: Synthetic Preferences for Self-Improving Reward Models
  8. 2025GRAM: A Generative Foundation Reward Model for Reward Generalization