mlsanityData to 7 Oct 2026

Aligning Large Language Models by On-Policy Self-Judgment

arXiv 2402.11253 ↗First posted 17 Feb 2024cs.LG cs.AI cs.CLcited by 20

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Existing approaches for aligning large language models with human preferences face a trade-off that requires a separate reward model (RM) for on-policy learning. In this paper, we present a novel alignment framework, SELF-JUDGE that (1) does on-policy learning and 2) is parameter efficient, as it does not require an additional RM for evaluating the samples for on-policy learning. To this end, we propose Judge-augmented Supervised Fine-Tuning (JSFT) to train a single model to act as both a policy and a judge. Specifically, we view the pairwise judgment task, choosing the better response from a response pair, as a special case of the instruction-following task. The resulting model can judge preferences of on-the-fly responses from current policy initialized from itself. Experimental results show the efficacy of SELF-JUDGE, outperforming baselines in preference benchmarks. We also show that the rejecting sampling by itself can improve performance further without an additional evaluator.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 27 of its 35 references that are in the map. First posted on arXiv 17 Feb 2024.

Builds on

27 of 35 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023Some things are more CRINGE than others: Iterative Preference Optimization with the Pairwise Cringe Loss27 Dec
  2. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  3. 2023Statistical Rejection Sampling Improves Preference Optimization13 Sep
  4. 2022Training language models to follow instructions with human feedback4 Mar
  5. 2021A General Language Assistant as a Laboratory for Alignment1 Dec

Other references · 22

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  7. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  8. 2020Learning to summarize from human feedbackcited by 2,562
  9. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  10. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  11. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  12. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 558
10 more
  1. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  2. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
  3. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
  4. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 288
  5. 2023Training Language Models with Language Feedback at Scalecited by 125
  6. 2017A Deeper Look at Experience Replaycited by 104
  7. 2023Navigating the Grey Area: How Expressions of Uncertainty and Overconfidence Affect Language Modelscited by 91
  8. 2023Human Feedback is not Gold Standardcited by 82
  9. 2023Loose lips sink ships: Mitigating Length Bias in Reinforcement Learning from Human Feedbackcited by 78
  10. 2023On the Calibration of Large Language Models and Alignmentcited by 72

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 10; 2026: 2.

20 papers.

Sort:

By area:

  1. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 480
  2. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 451
  3. 2025Evolving Language Models without Labels: Majority Drives Selection, Novelty Promotes Variationcited by 44
  4. 2024Provable Scaling Laws for the Test-Time Compute of Large Language Modelscited by 25
  5. 2025Online Preference Alignment for Language Models via Count-based Explorationcited by 24
  6. 2024Innovative Thinking, Infinite Humor: Humor Research of Large Language Models through Structured Thought Leapscited by 12
  7. 2024D2PO: Discriminator-Guided DPO with Response Evaluation Modelscited by 9
  8. 2024Direct Alignment of Language Models via Quality-Aware Self-Refinementcited by 6
  9. 2025Semantic Voting: A Self-Evaluation-Free Approach for Efficient LLM Self-Improvement on Unverifiable Open-ended Taskscited by 6
  10. 2025KL Penalty Control via Perturbation for Direct Preference Optimizationcited by 5
  11. 2025DecompileBench: A Comprehensive Benchmark for Evaluating Decompilers in Real-World Scenarioscited by 4
  12. 2025MedVAL: Toward Expert-Level Medical Text Validation with Language Modelscited by 3
8 more
  1. 2025Being Kind Isn't Always Being Safe: Diagnosing Affective Hallucination in LLMscited by 3
  2. 2025Do Before You Judge: Self-Reference as a Pathway to Better LLM Evaluationcited by 2
  3. 2025SGPO: Self-Generated Preference Optimization based on Self-Improvercited by 1
  4. 2026Simple Role Assignment is Extraordinarily Effective for Safety Alignmentcited by 1
  5. 2024Online Self-Preferring Language Modelscited by 0
  6. 2024Self-supervised Preference Optimization: Enhance Your Language Model with Preference Degree Awarenesscited by 0
  7. 2025Amulet: Putting Complex Multi-Turn Conversations on the Stand with LLM Juriescited by 0
  8. 2026RecurSE: Bounded Recursive Self-Evaluation for LLM Rubric Judgescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Self-Rewarding PPO: Aligning Large Language Models with Demonstrations Only
  2. 2026The Flip Side of RLHF: On-Policy Feedback for Reward Model Self-Supervised Improvement
  3. 2025Self-Consistency of the Internal Reward Models Improves Self-Rewarding Language Models
  4. 2025SIMPLEMIX: Frustratingly Simple Mixing of Off- and On-policy Data in Language Model Preference Learning
  5. 2024Self-Exploring Language Models: Active Preference Elicitation for Online Alignment
  6. 2024Fine-Tuning Language Models with Reward Learning on Policy
  7. 2024Reward-Augmented Data Enhances Direct Preference Alignment of LLMs
  8. 2025Aligning Large Language Models via Fully Self-Synthetic Data