mlsanityData to 7 Oct 2026

Direct Alignment of Language Models via Quality-Aware Self-Refinement

arXiv 2405.21040 ↗First posted 31 May 2024cs.CL cs.AIcited by 6

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Reinforcement Learning from Human Feedback (RLHF) has been commonly used to align the behaviors of Large Language Models (LLMs) with human preferences. Recently, a popular alternative is Direct Policy Optimization (DPO), which replaces an LLM-based reward model with the policy itself, thus obviating the need for extra memory and training time to learn the reward model. However, DPO does not consider the relative qualities of the positive and negative responses, and can lead to sub-optimal training outcomes. To alleviate this problem, we investigate the use of intrinsic knowledge within the on-the-fly fine-tuning LLM to obtain relative qualities and help to refine the loss function. Specifically, we leverage the knowledge of the LLM to design a refinement function to estimate the quality of both the positive and negative responses. We show that the constructed refinement function can help self-refine the loss function under mild assumptions. The refinement function is integrated into DPO and its variant Identity Policy Optimization (IPO). Experiments across various evaluators indicate that they can improve the performance of the fine-tuned models over DPO and IPO.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 37 of its 50 references that are in the map. First posted on arXiv 31 May 2024.

Builds on

37 of 50 references.

Closest ancestors · 11

Same direction, newest first

  1. 2024Understanding the performance gap between online and offline alignment algorithms14 May
  2. 2024Investigating Regularization of Self-Play Language Models4 Apr
  3. 2024Human Alignment of Large Language Models through Online Preference Optimisation13 Mar
  4. 2024Aligning Large Language Models by On-Policy Self-Judgment17 Feb
  5. 2024Direct Preference Optimization with an Offset16 Feb
  6. 2024ICDPO: Effectively Borrowing Alignment Capability of Others via In-context Direct Preference Optimization14 Feb
  7. 2024Aligner: Efficient Alignment by Learning to Correct4 Feb
  8. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  9. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints28 Sep
  10. 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision4 May
  11. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 26

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  9. 2017Deep reinforcement learning from human preferencescited by 3,846
  10. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  11. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  12. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
14 more
  1. 2023Mistral 7Bcited by 3,065
  2. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  3. 2023A Survey of Large Language Modelscited by 2,537
  4. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  5. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  6. 2024Self-Rewarding Language Modelscited by 654
  7. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 641
  8. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  9. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  10. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 431
  11. 2023AI Alignment: A Comprehensive Surveycited by 256
  12. 2023Nash Learning from Human Feedbackcited by 220
  13. 2024Self-playing Adversarial Language Game Enhances LLM Reasoningcited by 80
  14. 2024Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferencescited by 27

Built on by

Cited by year

Citing papers per year: 2024: 5; 2025: 1.

6 papers.

  1. 2025A Survey of Direct Preference Optimizationcited by 35
  2. 2024Towards a Unified View of Preference Learning for Large Language Models: A Surveycited by 17
  3. 2024Little Giants: Synthesizing High-Quality Embedding Data at Scalecited by 11
  4. 2024TODO: Enhancing LLM Alignment with Ternary Preferencescited by 5
  5. 2024REAL: Response Embedding-based Alignment for LLMscited by 4
  6. 2024Self-supervised Preference Optimization: Enhance Your Language Model with Preference Degree Awarenesscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025A Principled Loss Function for Direct Language Model Alignment
  2. 2024RS-DPO: A Hybrid Rejection Sampling and Direct Preference Optimization Method for Alignment of Large Language Models
  3. 2024Towards Efficient Exact Optimization of Language Model Alignment
  4. 2024Mixed Preference Optimization: Reinforcement Learning with Data Selection and Better Reference Model
  5. 2025Self-Rewarding PPO: Aligning Large Language Models with Demonstrations Only
  6. 2025AlignDistil: Token-Level Language Model Alignment as Adaptive Policy Distillation
  7. 2024Would I Lie To You? Inference Time Alignment of Language Models using Direct Preference Heads
  8. 2024Unified Preference Optimization: Language Model Alignment Beyond the Preference Frontier