mlsanityData to 7 Oct 2026

Improving Reward Models with Synthetic Critiques

arXiv 2405.20850 ↗First posted 31 May 2024cs.CLcited by 44

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reward models (RMs) play a critical role in aligning language models through the process of reinforcement learning from human feedback. RMs are trained to predict a score reflecting human preference, which requires significant time and cost for human annotation. Additionally, RMs tend to quickly overfit on superficial features in the training set, hindering their generalization performance on unseen distributions. We propose a novel approach using synthetic natural language critiques generated by large language models to provide additional feedback, evaluating aspects such as instruction following, correctness, and style. This offers richer signals and more robust features for RMs to assess and score on. We demonstrate that high-quality critiques improve the performance and data efficiency of RMs initialized from different pretrained models, reducing the reliance on costly human annotations. Furthermore, incorporating critiques improves both the interpretability and robustness of RM training.

Key facts. Cited by 44 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 38 of its 55 references that are in the map. First posted on arXiv 31 May 2024.

Builds on

38 of 55 references.

Closest ancestors · 12

Same direction, newest first

  1. 2023Generalization Analogies: A Testbed for Generalizing AI Oversight to Hard-To-Measure Domains13 Nov
  2. 2023Loose lips sink ships: Mitigating Length Bias in Reinforcement Learning from Human Feedback8 Oct
  3. 2023A Long Way to Go: Investigating Length Correlations in RLHF5 Oct
  4. 2023Reward Model Ensembles Help Mitigate Overoptimization4 Oct
  5. 2023The Trickle-down Impact of Reward (In-)consistency on RLHF28 Sep
  6. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback27 Jul
  7. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Training2 Jun
  8. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  9. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  10. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  11. 2019Fine-Tuning Language Models from Human Preferences18 Sep
  12. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 26

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  7. 2023Let's Verify Step by Stepcited by 3,598
  8. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  9. 2023Mistral 7Bcited by 3,065
  10. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  11. 2020Learning to summarize from human feedbackcited by 2,562
  12. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
14 more
  1. 2023Large Language Modelscited by 774
  2. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 751
  3. 2023Instruction Tuning for Large Language Models: A Surveycited by 617
  4. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 477
  5. 2022Self-critiquing models for assisting human evaluatorscited by 372
  6. 2023PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimizationcited by 311
  7. 2024Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Modelscited by 301
  8. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 290
  9. 2023OctoPack: Instruction Tuning Code Large Language Modelscited by 178
  10. 2023Do-Not-Answer: A Dataset for Evaluating Safeguards in LLMscited by 176
  11. 2023Generative Judge for Evaluating Alignmentcited by 163
  12. 2023Training Language Models with Language Feedback at Scalecited by 125
  13. 2023Human Feedback is not Gold Standardcited by 82
  14. 2024LLMCRIT: Teaching Large Language Models to Use Criteriacited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 10; 2025: 24; 2026: 10.

44 papers.

Sort:

By area:

  1. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 465
  2. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  3. 2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainscited by 328
  4. 2024Critique-out-Loud Reward Modelscited by 105
  5. 2025Learning to Plan & Reason for Evaluation with Thinking-LLM-as-a-Judgecited by 75
  6. 2025J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learningcited by 73
  7. 2025HelpSteer3-Preference: Open Human-Annotated Preference Data across Diverse Tasks and Languagescited by 71
  8. 2024Self-Generated Critiques Boost Reward Modeling for Language Modelscited by 69
  9. 2024TICKing All the Boxes: Generated Checklists Improve LLM Evaluation and Generationcited by 62
  10. 2026Rethinking Rubric Generation for Improving LLM Judge and Reward Modeling for Open-ended Taskscited by 55
  11. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  12. 2025A Survey of Test-Time Compute: From Intuitive Inference to Deliberate Reasoningcited by 40
32 more
  1. 2025Teaching Language Models to Critique via Reinforcement Learningcited by 38
  2. 2024Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Modelscited by 31
  3. 2024Beyond Scalar Reward Model: Learning Generative Judge from Preference Datacited by 30
  4. 2026Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challengescited by 30
  5. 2025R3: Robust Rubric-Agnostic Reward Modelscited by 22
  6. 2025Uncertainty-Aware Step-wise Verification with Generative Reward Modelscited by 21
  7. 2025Think-J: Learning to Think for Generative LLM-as-a-Judgecited by 20
  8. 2024Training Language Models to Critique With Multi-agent Feedbackcited by 16
  9. 2025HelpSteer3: Human-Annotated Feedback and Edit Data to Empower Inference-Time Scaling in Open-Ended General-Domain Taskscited by 16
  10. 2025Think-RM: Enabling Long-Horizon Reasoning in Generative Reward Modelscited by 15
  11. 2025Sailing by the Stars: A Survey on Reward Models and Learning Strategies for Learning from Rewardscited by 14
  12. 2026Outcome Accuracy is Not Enough: Aligning the Reasoning Process of Reward Modelscited by 9
  13. 2025LongDPO: Unlock Better Long-form Generation Abilities for LLMs via Critique-augmented Stepwise Informationcited by 8
  14. 2025mR3: Multilingual Rubric-Agnostic Reward Reasoning Modelscited by 8
  15. 2026Reward Modeling from Natural Language Human Feedbackcited by 7
  16. 2024Understanding Likelihood Over-optimisation in Direct Alignment Algorithmscited by 6
  17. 2025Align to Structure: Aligning Large Language Models with Structural Informationcited by 5
  18. 2025J1: Exploring Simple Test-Time Scaling for LLM-as-a-Judgecited by 5
  19. 2025S2J: Bridging the Gap Between Solving and Judging Ability in Generative Reward Modelscited by 5
  20. 2025PaTaRM: Bridging Pairwise and Pointwise Signals via Preference-Aware Task-Adaptive Reward Modelingcited by 5
  21. 2024Data-Centric Human Preference with Rationales for Direct Preference Alignmentcited by 4
  22. 2025RPO: Fine-Tuning Visual Generative Models via Rich Vision-Language Preferencescited by 4
  23. 2024The Superalignment of Superhuman Intelligence with Large Language Modelscited by 2
  24. 2025Leveraging Robust Optimization for LLM Alignment under Distribution Shiftscited by 2
  25. 2025TRPrompt: Bootstrapping Query-Aware Prompt Optimization from Textual Rewardscited by 2
  26. 2025Fine-Tuning LLMs with Fine-Grained Human Feedback on Text Spanscited by 1
  27. 2026RM-Distiller: Exploiting Generative LLM for Reward Model Distillationcited by 1
  28. 2026Lattice: Generative Guardrails for Conversational Agentscited by 1
  29. 2026Learning Ordinal Probabilistic Reward from Preferencescited by 1
  30. 2026Towards Better RL Training Data Utilization via Second-Order Rolloutcited by 1
  31. 2026ConsistRM: Improving Generative Reward Models via Consistency-Aware Self-Trainingcited by 1
  32. 2026RL with Learnable Textual Feedback: A Bilevel Approachcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024West-of-N: Synthetic Preferences for Self-Improving Reward Models
  2. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style
  3. 2024Beyond Sparse Rewards: Enhancing Reinforcement Learning with Language Model Critique in Text Generation
  4. 2025GRAM: A Generative Foundation Reward Model for Reward Generalization
  5. 2024RewardBench: Evaluating Reward Models for Language Modeling
  6. 2025A Systematic Analysis of Base Model Choice for Reward Modeling
  7. 2023Aligning Large Language Models through Synthetic Feedback
  8. 2024Sequence to Sequence Reward Modeling: Improving RLHF by Language Feedback