mlsanityData to 14 Oct 2026

Beyond Scalar Reward Model: Learning Generative Judge from Preference Data

arXiv 2410.03742 ↗First posted 1 Oct 2024cs.CL cs.AI cs.LGcited by 32

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Learning from preference feedback is a common practice for aligning large language models~(LLMs) with human value. Conventionally, preference data is learned and encoded into a scalar reward model that connects a value head with an LLM to produce a scalar score as preference or reward. However, scalar models lack interpretability and are known to be susceptible to biases in datasets. This paper investigates leveraging the generation capability of LLMs to address both limitations in one shot. Specifically, we prompt the pre-trained LLM to generate positive and negative judgments, both supported with rationales in natural language form. The self-generated contrastive judgment pairs are used to train the generative judge with Direct Preference Optimization (DPO). This proposal of training the generative Judge using self-generated Contrastive judgments (Con-J) ensures natural interpretability due to the generated rationales together with the judgments, as well as high robustness against bias without the need for an additional reward head. Experimental results show that the performance of Con-J is comparable to the scalar reward model trained on the same collection of preference data, and demonstrate its superior interpretability and robustness in encoding human preferences.

Key facts. Cited by 32 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 33 of its 40 references that are in the map. First posted on arXiv 1 Oct 2024.

Builds on

33 of 40 references.

Closest ancestors · 12

Same direction, newest first

  1. 2024Critique-out-Loud Reward Models21 Aug
  2. 2024Improving Reward Models with Synthetic Critiques31 May
  3. 2024Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizer26 May
  4. 2024RewardBench: Evaluating Reward Models for Language Modeling20 Mar
  5. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint18 Dec
  6. 2023Loose lips sink ships: Mitigating Length Bias in Reinforcement Learning from Human Feedback8 Oct
  7. 2023A Long Way to Go: Investigating Length Correlations in RLHF5 Oct
  8. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback27 Jul
  9. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  10. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedback17 May
  11. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  12. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 21

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  3. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  4. 2020Learning to summarize from human feedbackcited by 2,588
  5. 2024Qwen2 Technical Reportcited by 2,120
  6. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  7. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 912
  8. 2016Training Deep Nets with Sublinear Memory Costcited by 848
  9. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 564
  10. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 562
  11. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 480
  12. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 446
9 more
  1. 2024Direct Language Model Alignment from Online AI Feedbackcited by 260
  2. 2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positivecited by 241
  3. 2024Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judgecited by 204
  4. 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Setscited by 201
  5. 2023Generative Judge for Evaluating Alignmentcited by 169
  6. 2023Some things are more CRINGE than others: Iterative Preference Optimization with the Pairwise Cringe Losscited by 108
  7. 2024OffsetBias: Leveraging Debiased Data for Tuning Evaluatorscited by 94
  8. 2023Applying Large Language Models and Chain-of-Thought for Automatic Scoringcited by 69
  9. 2024ChatGLM-RLHF: Practices of Aligning Large Language Models with Human Feedbackcited by 25

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 25; 2026: 5.

32 papers.

Sort:

By area:

  1. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 465
  2. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 111
  3. 2024Reinforcement Learning Enhanced LLMs: A Surveycited by 71
  4. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  5. 2025Writing-Zero: Bridge the Gap Between Non-verifiable Tasks and Verifiable Rewardscited by 50
  6. 2025Evaluating Judges as Evaluators: The JETTS Benchmark of LLM-as-Judges as Test-Time Scaling Evaluatorscited by 44
  7. 2025Improve LLM-as-a-Judge Ability as a General Abilitycited by 37
  8. 2025Improving LLM-as-a-Judge Inference with the Judgment Distributioncited by 37
  9. 2026Open Rubric System: Scaling Reinforcement Learning with Pairwise Adaptive Rubriccited by 27
  10. 2025Does Context Matter? ContextualJudgeBench for Evaluating LLM-based Judges in Contextual Settingscited by 24
  11. 2025IPO: Your Language Model is Secretly a Preference Classifiercited by 17
  12. 2025Beyond the Surface: Enhancing LLM-as-a-Judge Alignment with Human via Internal Representationscited by 15
20 more
  1. 2025Sailing by the Stars: A Survey on Reward Models and Learning Strategies for Learning from Rewardscited by 14
  2. 2025Generative RLHF-V: Learning Principles from Multi-modal Human Preferencecited by 13
  3. 2025CompassJudger-2: Towards Generalist Judge Model via Verifiable Rewardscited by 12
  4. 2026Reward Modeling from Natural Language Human Feedbackcited by 11
  5. 2025J4R: Learning to Judge with Equivalent Initial State Group Relative Policy Optimizationcited by 9
  6. 2025GenCLS++: Pushing the Boundaries of Generative Classification in LLMs Through Comprehensive SFT and RL Studies Across Diverse Datasetscited by 6
  7. 2025S2J: Bridging the Gap Between Solving and Judging Ability in Generative Reward Modelscited by 5
  8. 2025LongRM: Revealing and Unlocking the Context Boundary of Reward Modelingcited by 5
  9. 2025Foundational Automatic Evaluators: Scaling Multi-Task Generative Evaluator Training for Reasoning-Centric Domainscited by 5
  10. 2025Assistant-Guided Mitigation of Teacher Preference Bias in LLM-as-a-Judgecited by 3
  11. 2025On the Shelf Life of Fine-Tuned LLM-Judges: Future-Proofing, Backward-Compatibility, and Question Generalizationcited by 3
  12. 2025Probing Preference Representations: A Multi-Dimensional Evaluation and Analysis Method for Reward Modelscited by 3
  13. 2025JudgeAgent: Beyond Static Benchmarks for Knowledge-Driven and Dynamic LLM Evaluationcited by 2
  14. 2025Better Language Model-Based Judging Reward Modeling through Scaling Comprehension Boundariescited by 1
  15. 2025RAGferee: Building Contextual Reward Models for Retrieval-Augmented Generationcited by 1
  16. 2025PIRA: Preference-Oriented Instruction-Tuned Reward Models with Dual Aggregationcited by 1
  17. 2025GFRIEND: Generative Few-shot Reward Inference through EfficieNt DPOcited by 0
  18. 2026Generate to Explore, Select to Exploit: Aligning LLM-based Headline Generation with Personalized Recommendationcited by 0
  19. 2026EnGRICH: Enhancing Generative Reward Modeling with Critiques from Humanscited by 0
  20. 2026RM-OPD: On-Policy Distillation for Generative Reward Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Direct Judgement Preference Optimization
  2. 2025Think-J: Learning to Think for Generative LLM-as-a-Judge
  3. 2026RM-Distiller: Exploiting Generative LLM for Reward Model Distillation
  4. 2026ReflectRM: Boosting Generative Reward Models via Self-Reflection within a Unified Judgment Framework
  5. 2024Generative Reward Models
  6. 2024Boosting Reward Model with Preference-Conditional Multi-Aspect Synthetic Data Generation
  7. 2025Limited Preference Data? Learning Better Reward Model with Latent Space Synthesis
  8. 2026Preference learning in shades of gray: Interpretable and bias-aware reward modeling for human preferences