mlsanityData to 14 Oct 2026

Critique-out-Loud Reward Models

arXiv 2408.11791 ↗First posted 21 Aug 2024cs.LGcited by 106

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Traditionally, reward models used for reinforcement learning from human feedback (RLHF) are trained to directly predict preference scores without leveraging the generation capabilities of the underlying large language model (LLM). This limits the capabilities of reward models as they must reason implicitly about the quality of a response, i.e., preference modeling must be performed in a single forward pass through the model. To enable reward models to reason explicitly about the quality of a response, we introduce Critique-out-Loud (CLoud) reward models. CLoud reward models operate by first generating a natural language critique of the assistant's response that is then used to predict a scalar reward for the quality of the response. We demonstrate the success of CLoud reward models for both Llama-3-8B and 70B base models: compared to classic reward models CLoud reward models improve pairwise preference classification accuracy on RewardBench by 4.65 and 5.84 percentage points for the 8B and 70B base models respectively. Furthermore, CLoud reward models lead to a Pareto improvement for win rate on ArenaHard when used as the scoring model for Best-of-N. Finally, we explore how to exploit the dynamic inference compute capabilities of CLoud reward models by performing self-consistency decoding for reward prediction.

Key facts. Cited by 106 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 45 of its 52 references that are in the map. First posted on arXiv 21 Aug 2024.

Builds on

45 of 52 references.

Closest ancestors · 14

Same direction, newest first

  1. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts18 Jun
  2. 2024Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMs14 Jun
  3. 2024HelpSteer2: Open-source dataset for training top-performing reward models12 Jun
  4. 2024Improving Reward Models with Synthetic Critiques31 May
  5. 2024RLHF Workflow: From Reward Modeling to Online RLHF13 May
  6. 2024RewardBench: Evaluating Reward Models for Language Modeling20 Mar
  7. 2024Self-Rewarding Language Models18 Jan
  8. 2024A Minimaximalist Approach to Reinforcement Learning from Human Feedback8 Jan
  9. 2023Nash Learning from Human Feedback1 Dec
  10. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback1 Sep
  11. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedback17 May
  12. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons26 Jan
2 more
  1. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  2. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 31

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  6. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  7. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  8. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  9. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  10. 2020Learning to summarize from human feedbackcited by 2,588
  11. 2023Teaching Large Language Models to Self-Debugcited by 988
  12. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 870
19 more
  1. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 783
  2. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
  3. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 575
  4. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 564
  5. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 500
  6. 2022Self-critiquing models for assisting human evaluatorscited by 380
  7. 2023Language Models can Solve Computer Taskscited by 372
  8. 2023Statistical Rejection Sampling Improves Preference Optimizationcited by 323
  9. 2022RARR: Researching and Revising What Language Models Say, Using Language Modelscited by 287
  10. 2024Advancing LLM Reasoning Generalists with Preference Treescited by 192
  11. 2023The Capacity for Moral Self-Correction in Large Language Modelscited by 173
  12. 2023Generative Judge for Evaluating Alignmentcited by 169
  13. 2023Is ChatGPT better than Human Annotators? Potential and Limitations of ChatGPT in Explaining Implicit Hate Speechcited by 166
  14. 2024LLM Critics Help Catch LLM Bugscited by 147
  15. 2023HelpSteer: Multi-attribute Helpfulness Dataset for SteerLMcited by 142
  16. 2023Training Language Models with Language Feedback at Scalecited by 128
  17. 2023RL4F: Generating Natural Language Feedback with Reinforcement Learning for Repairing Model Outputscited by 105
  18. 2017Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedbackcited by 72
  19. 2024Self-Refinement of Language Models from External Proxy Metrics Feedbackcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 10; 2025: 68; 2026: 28.

106 papers.

Sort:

By area:

  1. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,210
  2. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 480
  3. 2025Inference-Time Scaling for Generalist Reward Modelingcited by 239
  4. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  5. 2025AIMO-2 Winning Solution: Building State-of-the-Art Mathematical Reasoning Models with OpenMathReasoning datasetcited by 174
  6. 2025RewardBench 2: Advancing Reward Model Evaluationcited by 150
  7. 2025OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignmentcited by 132
  8. 2025Process Reward Models That Thinkcited by 118
  9. 2025GenPRM: Scaling Test-Time Compute of Process Reward Models via Generative Reasoningcited by 97
  10. 2025Learning to Plan & Reason for Evaluation with Thinking-LLM-as-a-Judgecited by 78
  11. 2025J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learningcited by 75
  12. 2025HelpSteer3-Preference: Open Human-Annotated Preference Data across Diverse Tasks and Languagescited by 75
38 more
  1. 2024Self-Generated Critiques Boost Reward Modeling for Language Modelscited by 71
  2. 2024Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Modelscited by 70
  3. 2024TICKing All the Boxes: Generated Checklists Improve LLM Evaluation and Generationcited by 68
  4. 2024Adaptive Inference-Time Compute: LLMs Can Predict if They Can Do Better, Even Mid-Generationcited by 64
  5. 2025Critique Fine-Tuning: Learning to Critique is More Effective than Learning to Imitatecited by 63
  6. 2024RMB: Comprehensively Benchmarking Reward Models in LLM Alignmentcited by 61
  7. 2025Reinforcement Learning from Human Feedbackcited by 58
  8. 2026Alternating Reinforcement Learning for Rubric-Based Reward Modeling in Non-Verifiable LLM Post-Trainingcited by 54
  9. 2025A Survey on Large Language Models for Mathematical Reasoningcited by 52
  10. 2025DeSTA2.5-Audio: Toward General-Purpose Large Audio Language Model with Self-Generated Cross-Modal Alignmentcited by 50
  11. 2025ReasonFlux-PRM: Trajectory-Aware PRMs for Long Chain-of-Thought Reasoning in LLMscited by 46
  12. 2025Teaching Language Models to Critique via Reinforcement Learningcited by 42
  13. 2025When To Solve, When To Verify: Compute-Optimal Problem Solving and Generative Verification for LLM Reasoningcited by 42
  14. 2025A Survey of Test-Time Compute: From Intuitive Inference to Deliberate Reasoningcited by 40
  15. 2025Do LLM Evaluators Prefer Themselves for a Reason?cited by 38
  16. 2025Harnessing the Reasoning Economy: A Survey of Efficient Reasoning for Large Language Modelscited by 36
  17. 2024Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Modelscited by 35
  18. 2024Beyond Scalar Reward Model: Learning Generative Judge from Preference Datacited by 32
  19. 2025Two Heads are Better Than One: Test-time Scaling of Multi-agent Collaborative Reasoningcited by 29
  20. 2025RewardAnything: Generalizable Principle-Following Reward Modelscited by 28
  21. 2025To Backtrack or Not to Backtrack: When Sequential Search Limits Model Reasoningcited by 24
  22. 2025R3: Robust Rubric-Agnostic Reward Modelscited by 23
  23. 2025Fast Controlled Generation from Language Models with Adaptive Weighted Rejection Samplingcited by 21
  24. 2025Revisiting Multi-Agent Debate as Test-Time Scaling: A Systematic Study of Conditional Effectivenesscited by 20
  25. 2024Mind the Gap: Examining the Self-Improvement Capabilities of Large Language Modelscited by 18
  26. 2025Language Models Can Learn from Verbal Feedback Without Scalar Rewardscited by 18
  27. 2025HelpSteer3: Human-Annotated Feedback and Edit Data to Empower Inference-Time Scaling in Open-Ended General-Domain Taskscited by 17
  28. 2025TRACT: Regression-Aware Fine-tuning Meets Chain-of-Thought Reasoning for LLM-as-a-Judgecited by 17
  29. 2025Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learningcited by 17
  30. 2025Reasoning Beyond Limits: Advances and Open Problems for LLMscited by 16
  31. 2025Think-RM: Enabling Long-Horizon Reasoning in Generative Reward Modelscited by 16
  32. 2025CodeCriticBench: A Holistic Code Critique Benchmark for Large Language Modelscited by 14
  33. 2025Putting the Value Back in RL: Better Test-Time Scaling by Unifying LLM Reasoners With Verifierscited by 14
  34. 2025Enhancing Mathematical Reasoning in Large Language Models with Self-Consistency-Based Hallucination Detectioncited by 13
  35. 2025Generative RLHF-V: Learning Principles from Multi-modal Human Preferencecited by 13
  36. 2025Good Learners Think Their Thinking: Generative PRM Makes Large Reasoning Model More Efficient Math Learnercited by 11
  37. 2026Reward Modeling from Natural Language Human Feedbackcited by 11
  38. 2024CodePMP: Scalable Preference Model Pretraining for Large Language Model Reasoningcited by 10

50 most cited of 106 All 106 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024How to Evaluate Reward Models for RLHF
  2. 2024Secrets of RLHF in Large Language Models Part II: Reward Modeling
  3. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  4. 2025A Systematic Analysis of Base Model Choice for Reward Modeling
  5. 2025Better Language Model-Based Judging Reward Modeling through Scaling Comprehension Boundaries
  6. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style
  7. 2025A Unified Pairwise Framework for RLHF: Bridging Generative Reward Modeling and Policy Optimization
  8. 2025Off-Policy Corrected Reward Modeling for Reinforcement Learning from Human Feedback