mlsanityData to 7 Oct 2026

Self-Generated Critiques Boost Reward Modeling for Language Models

arXiv 2411.16646 ↗First posted 25 Nov 2024cs.CL cs.AI cs.LGcited by 69

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 2 more
Abstract (arXiv)

Reward modeling is crucial for aligning large language models (LLMs) with human preferences, especially in reinforcement learning from human feedback (RLHF). However, current reward models mainly produce scalar scores and struggle to incorporate critiques in a natural language format. We hypothesize that predicting both critiques and the scalar reward would improve reward modeling ability. Motivated by this, we propose Critic-RM, a framework that improves reward models using self-generated critiques without extra supervision. Critic-RM employs a two-stage process: generating and filtering high-quality critiques, followed by joint fine-tuning on reward prediction and critique generation. Experiments across benchmarks show that Critic-RM improves reward modeling accuracy by 3.7%-7.3% compared to standard reward models and LLM judges, demonstrating strong performance and data efficiency. Additional studies further validate the effectiveness of generated critiques in rectifying flawed reasoning steps with 2.5%-3.2% gains in improving reasoning accuracy.

Key facts. Cited by 69 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 63 of its 71 references that are in the map. First posted on arXiv 25 Nov 2024.

Builds on

63 of 71 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Generative Verifiers: Reward Modeling as Next-Token Prediction27 Aug
  2. 2024Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback13 Jun

Other references · 61

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2024The Llama 3 Herd of Modelscited by 14,192
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  7. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  8. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  9. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  10. 2024GPT-4o System Cardcited by 4,467
  11. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  12. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
38 more
  1. 2020Learning to summarize from human feedbackcited by 2,562
  2. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
  3. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
  4. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  5. 2024Self-Rewarding Language Modelscited by 654
  6. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  7. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
  8. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 558
  9. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
  10. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 480
  11. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 431
  12. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 399
  13. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 394
  14. 2022Self-critiquing models for assisting human evaluatorscited by 372
  15. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 369
  16. 2022Large Language Models Are Reasoning Teacherscited by 318
  17. 2024Humans or LLMs as the Judge? A Study on Judgement Biasescited by 314
  18. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 290
  19. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 288
  20. 2023A Long Way to Go: Investigating Length Correlations in RLHFcited by 285
  21. 2024Iterative Reasoning Preference Optimizationcited by 244
  22. 2024LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Taskscited by 236
  23. 2024HelpSteer2: Open-source dataset for training top-performing reward modelscited by 232
  24. 2023Proving Test Set Contamination in Black Box Language Modelscited by 227
  25. 2023Reward Model Ensembles Help Mitigate Overoptimizationcited by 225
  26. 2024Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judgecited by 202
  27. 2023Generative Judge for Evaluating Alignmentcited by 163
  28. 2023Self-Alignment with Instruction Backtranslationcited by 149
  29. 2024Generative Reward Modelscited by 148
  30. 2024Large Language Models are Inconsistent and Biased Evaluatorscited by 146
  31. 2024WARM: On the Benefits of Weight Averaged Reward Modelscited by 145
  32. 2024Nemotron-4 340B Technical Reportcited by 143
  33. 2023HelpSteer: Multi-attribute Helpfulness Dataset for SteerLMcited by 140
  34. 2024LLM Critics Help Catch LLM Bugscited by 138
  35. 2024ODIN: Disentangled Reward Mitigates Hacking in RLHFcited by 133
  36. 2024HelpSteer2-Preference: Complementing Ratings with Preferencescited by 129
  37. 2024Critique-out-Loud Reward Modelscited by 105
  38. 2024CriticBench: Benchmarking LLMs for Critique-Correct Reasoningcited by 88

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 46; 2026: 21.

69 papers.

Sort:

By area:

  1. 2024Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMscited by 564
  2. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  3. 2025RM-R1: Reward Modeling as Reasoningcited by 149
  4. 2025RewardBench 2: Advancing Reward Model Evaluationcited by 136
  5. 2025A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyondcited by 131
  6. 2025OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignmentcited by 118
  7. 2025MM-RLHF: The Next Step Forward in Multimodal LLM Alignmentcited by 79
  8. 2024AceMath: Advancing Frontier Math Reasoning with Post-Training and Reward Modelingcited by 76
  9. 2025Learning to Plan & Reason for Evaluation with Thinking-LLM-as-a-Judgecited by 75
  10. 2025J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learningcited by 73
  11. 2025HelpSteer3-Preference: Open Human-Annotated Preference Data across Diverse Tasks and Languagescited by 71
  12. 2025R1-Reward: Training Multimodal Reward Model Through Stable Reinforcement Learningcited by 65
38 more
  1. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  2. 2025Agentic Reward Modeling: Integrating Human Preferences with Verifiable Correctness Signals for Reliable Reward Systemscited by 49
  3. 2026Alternating Reinforcement Learning for Rubric-Based Reward Modeling in Non-Verifiable LLM Post-Trainingcited by 45
  4. 2025Feedback Descent: Open-Ended Text Optimization via Pairwise Comparisoncited by 42
  5. 2025A Survey of Test-Time Compute: From Intuitive Inference to Deliberate Reasoningcited by 40
  6. 2025SPC: Evolving Self-Play Critic via Adversarial Games for LLM Reasoningcited by 40
  7. 2025Teaching Language Models to Critique via Reinforcement Learningcited by 38
  8. 2025MCTS-Judge: Test-Time Scaling in LLM-as-a-Judge for Code Correctness Evaluationcited by 32
  9. 2026Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challengescited by 30
  10. 2025ThinkGuard: Deliberative Slow Thinking Leads to Cautious Guardrailscited by 29
  11. 2025RewardAnything: Generalizable Principle-Following Reward Modelscited by 27
  12. 2025Heimdall: test-time scaling on the generative verificationcited by 26
  13. 2025HelpSteer3: Human-Annotated Feedback and Edit Data to Empower Inference-Time Scaling in Open-Ended General-Domain Taskscited by 16
  14. 2025Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learningcited by 16
  15. 2025Self-Improvement Towards Pareto Optimality: Mitigating Preference Conflicts in Multi-Objective Alignmentcited by 15
  16. 2025Think-RM: Enabling Long-Horizon Reasoning in Generative Reward Modelscited by 15
  17. 2025BaseReward: A Strong Baseline for Multimodal Reward Modelcited by 15
  18. 2025Sailing by the Stars: A Survey on Reward Models and Learning Strategies for Learning from Rewardscited by 14
  19. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challengescited by 14
  20. 2025Ask a Strong LLM Judge when Your Reward Model is Uncertaincited by 13
  21. 2026CDRRM: Contrast-Driven Rubric Generation for Reliable and Interpretable Reward Modelingcited by 10
  22. 2025LongDPO: Unlock Better Long-form Generation Abilities for LLMs via Critique-augmented Stepwise Informationcited by 8
  23. 2025LatentEvolve: Self-Evolving Test-Time Scaling in Latent Spacecited by 8
  24. 2026Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learningcited by 8
  25. 2025CoT-based Synthesizer: Enhancing LLM Performance through Answer Synthesiscited by 7
  26. 2025DeCoRL: Decoupling Reasoning Chains via Parallel Sub-Step Generation and Cascaded Reinforcement for Interpretable and Scalable RLHFcited by 7
  27. 2026Reward Modeling from Natural Language Human Feedbackcited by 7
  28. 2025Higher Satisfaction, Lower Cost: A Technical Report on How LLMs Revolutionize Meituan's Intelligent Interaction Systemscited by 6
  29. 2026Reason Only When Needed: Efficient Generative Reward Modeling via Model-Internal Uncertaintycited by 6
  30. 2025J1: Exploring Simple Test-Time Scaling for LLM-as-a-Judgecited by 5
  31. 2025S2J: Bridging the Gap Between Solving and Judging Ability in Generative Reward Modelscited by 5
  32. 2026Deep Search with Hierarchical Meta-Cognitive Monitoring Inspired by Cognitive Neurosciencecited by 5
  33. 2025Scalable Oversight for Superhuman AI via Recursive Self-Critiquingcited by 4
  34. 2025ToolRM: Towards Agentic Tool-Use Reward Modelingcited by 4
  35. 2026CAMEL: Confidence-Gated Reflection for Reward Modelingcited by 3
  36. 2026Building a Precise Video Language with Human-AI Oversightcited by 3
  37. 2026RUBRIC-ARROW: Alternating Pointwise Rubric Reward Modeling for LLM Post-training in Non-verifiable Domainscited by 3
  38. 2026Skill-RM: Unifying Heterogeneous Evaluation Criteria via Agent Skillcited by 3

50 most cited of 69 All 69 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style
  2. 2024Beyond Sparse Rewards: Enhancing Reinforcement Learning with Language Model Critique in Text Generation
  3. 2025Better Language Model-Based Judging Reward Modeling through Scaling Comprehension Boundaries
  4. 2019Fine-Tuning Language Models from Human Preferences
  5. 2025Critique-RL: Training Language Models for Critiquing through Two-Stage Reinforcement Learning
  6. 2026Small Reward Models via Backward Inference
  7. 2024M-RewardBench: Evaluating Reward Models in Multilingual Settings
  8. 2025GRAM: A Generative Foundation Reward Model for Reward Generalization