mlsanityData to 7 Oct 2026

Direct Judgement Preference Optimization

arXiv 2409.14664 ↗First posted 23 Sep 2024cs.CLcited by 33

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Auto-evaluation is crucial for assessing response quality and offering feedback for model development. Recent studies have explored training large language models (LLMs) as generative judges to evaluate and critique other models' outputs. In this work, we investigate the idea of learning from both positive and negative data with preference optimization to enhance the evaluation capabilities of LLM judges across an array of different use cases. We achieve this by employing three approaches to collect the preference pairs for different use cases, each aimed at improving our generative judge from a different perspective. Our comprehensive study over a wide range of benchmarks demonstrates the effectiveness of our method. In particular, our generative judge achieves the best performance on 10 out of 13 benchmarks, outperforming strong baselines like GPT-4o and specialized judge models. Further analysis show that our judge model robustly counters inherent biases such as position and length bias, flexibly adapts to any evaluation protocol specified by practitioners, and provides helpful language feedback for improving downstream generator models.

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 62 of its 79 references that are in the map. First posted on arXiv 23 Sep 2024.

Builds on

62 of 79 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024LLM Evaluators Recognize and Favor Their Own Generations15 Apr
  2. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Models12 Oct

Other references · 60

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  8. 2023Let's Verify Step by Stepcited by 3,598
  9. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  10. 2024Mixtral of Expertscited by 1,785
  11. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  12. 2023Large Language Models are not Fair Evaluatorscited by 1,050
38 more
  1. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  2. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  3. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
  4. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  5. 2024Self-Rewarding Language Modelscited by 654
  6. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 550
  7. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 480
  8. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 431
  9. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 420
  10. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 399
  11. 2024InternLM2 Technical Reportcited by 391
  12. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 369
  13. 2023GPTScore: Evaluate as You Desirecited by 357
  14. 2023PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimizationcited by 311
  15. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 290
  16. 2023RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Modelscited by 272
  17. 2024RLHF Workflow: From Reward Modeling to Online RLHFcited by 265
  18. 2024MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documentscited by 251
  19. 2024Iterative Reasoning Preference Optimizationcited by 244
  20. 2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positivecited by 239
  21. 2024HelpSteer2: Open-source dataset for training top-performing reward modelscited by 232
  22. 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Setscited by 198
  23. 2021Muppet: Massive Multi-task Representations with Pre-Finetuningcited by 192
  24. 2023Benchmarking Foundation Models with Language-Model-as-an-Examinercited by 172
  25. 2023Generative Judge for Evaluating Alignmentcited by 163
  26. 2023Chain of Hindsight Aligns Language Models with Feedbackcited by 147
  27. 2023Benchmarking Cognitive Biases in Large Language Models as Evaluatorscited by 147
  28. 2024Nemotron-4 340B Technical Reportcited by 143
  29. 2024Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMscited by 142
  30. 2023HelpSteer: Multi-attribute Helpfulness Dataset for SteerLMcited by 140
  31. 2024InFoBench: Evaluating Instruction Following Ability in Large Language Modelscited by 128
  32. 2023A Critical Evaluation of Evaluations for Long-form Question Answeringcited by 111
  33. 2023RL4F: Generating Natural Language Feedback with Reinforcement Learning for Repairing Model Outputscited by 104
  34. 2023Shepherd: A Critic for Language Model Generationcited by 96
  35. 2024OffsetBias: Leveraging Debiased Data for Tuning Evaluatorscited by 94
  36. 2024The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Modelscited by 89
  37. 2024Foundational Autoraters: Taming Large Language Models for Better Automatic Evaluationcited by 72
  38. 2023Benchmarking Generation and Evaluation Capabilities of Large Language Models for Instruction Controllable Summarizationcited by 69

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 17; 2026: 8.

33 papers.

Sort:

By area:

  1. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  2. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMscited by 322
  3. 2025AIMO-2 Winning Solution: Building State-of-the-Art Mathematical Reasoning Models with OpenMathReasoning datasetcited by 165
  4. 2024HelpSteer2-Preference: Complementing Ratings with Preferencescited by 129
  5. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 108
  6. 2025Learning to Plan & Reason for Evaluation with Thinking-LLM-as-a-Judgecited by 75
  7. 2025J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learningcited by 73
  8. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  9. 2025Evaluating Judges as Evaluators: The JETTS Benchmark of LLM-as-Judges as Test-Time Scaling Evaluatorscited by 42
  10. 2024A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applicationscited by 27
  11. 2024Better Instruction-Following Through Minimum Bayes Riskcited by 19
  12. 2025Crowd Comparative Reasoning: Unlocking Comprehensive Evaluations for LLM-as-a-Judgecited by 18
21 more
  1. 2025Variation in Verification: Understanding Verification Dynamics in Large Language Modelscited by 18
  2. 2024Training Language Models to Critique With Multi-agent Feedbackcited by 16
  3. 2025Reasoning Beyond Limits: Advances and Open Problems for LLMscited by 16
  4. 2024GLIDER: Grading LLM Interactions and Decisions using Explainable Rankingcited by 14
  5. 2025J4R: Learning to Judge with Equivalent Initial State Group Relative Policy Optimizationcited by 9
  6. 2025Echoing: Identity Failures when LLM Agents Talk to Each Othercited by 8
  7. 2026Reward Modeling from Natural Language Human Feedbackcited by 7
  8. 2025Light-IF: Endowing LLMs with Generalizable Reasoning via Preview and Self-Checking for Complex Instruction Followingcited by 6
  9. 2025S2J: Bridging the Gap Between Solving and Judging Ability in Generative Reward Modelscited by 5
  10. 2025Foundational Automatic Evaluators: Scaling Multi-Task Generative Evaluator Training for Reasoning-Centric Domainscited by 5
  11. 2026Is Agentic RAG worth it? An experimental comparison of RAG approachescited by 5
  12. 2025IF-CRITIC: Towards a Fine-Grained LLM Critic for Instruction-Following Evaluationcited by 4
  13. 2025On the Shelf Life of Fine-Tuned LLM-Judges: Future-Proofing, Backward-Compatibility, and Question Generalizationcited by 3
  14. 2026FormalRewardBench: A Benchmark for Formal Theorem Proving Reward Modelscited by 2
  15. 2024Dr. SoW: Density Ratio of Strong-over-weak LLMs for Reducing the Cost of Human Annotation in Preference Tuningcited by 1
  16. 2026Comparing Developer and LLM Biases in Code Evaluationcited by 1
  17. 2026Jury Duty: Calibration and Orientation Failures in MLLM-as-a-Judge Under Cultural Ambiguitycited by 1
  18. 2024Optimizing Alignment with Less: Leveraging Data Augmentation for Personalized Evaluationcited by 0
  19. 2026Distributed Direct Preference Optimizationcited by 0
  20. 2026Counsel: A Meta-Evaluation Dataset for Agentic Taskscited by 0
  21. 2026Building Legal Reward Models for Grounding and Abstentioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Improve LLM-as-a-Judge Ability as a General Ability
  2. 2024Beyond Scalar Reward Model: Learning Generative Judge from Preference Data
  3. 2024The Fellowship of the LLMs: Multi-Model Workflows for Synthetic Preference Optimization Dataset Generation
  4. 2026JudgeArena: A Unified Framework for Reproducible LLM-Judge Evaluation
  5. 2026Toward Robust LLM-Based Judges: Taxonomic Bias Evaluation and Debiasing Optimization
  6. 2025Reverse Engineering Human Preferences with Reinforcement Learning
  7. 2025Think-J: Learning to Think for Generative LLM-as-a-Judge
  8. 2025Refine-n-Judge: Curating High-Quality Preference Chains for LLM-Fine-Tuning