mlsanityData to 7 Oct 2026

Direct Language Model Alignment from Online AI Feedback

arXiv 2402.04792 ↗First posted 7 Feb 2024cs.AI cs.CL cs.HCcited by 257

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Direct alignment from preferences (DAP) methods, such as DPO, have recently emerged as efficient alternatives to reinforcement learning from human feedback (RLHF), that do not require a separate reward model. However, the preference datasets used in DAP methods are usually collected ahead of training and never updated, thus the feedback is purely offline. Moreover, responses in these datasets are often sampled from a language model distinct from the one being aligned, and since the model evolves over training, the alignment phase is inevitably off-policy. In this study, we posit that online feedback is key and improves DAP methods. Our method, online AI feedback (OAIF), uses an LLM as annotator: on each training iteration, we sample two responses from the current model and prompt the LLM annotator to choose which one is preferred, thus providing online feedback. Despite its simplicity, we demonstrate via human evaluation in several tasks that OAIF outperforms both offline DAP and RLHF methods. We further show that the feedback leveraged in OAIF is easily controllable, via instruction prompts to the LLM annotator.

Key facts. Cited by 257 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 27 of its 32 references that are in the map. First posted on arXiv 7 Feb 2024.

Builds on

27 of 32 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023Some things are more CRINGE than others: Iterative Preference Optimization with the Pairwise Cringe Loss27 Dec
  2. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  3. 2023SALMON: Self-Alignment with Instructable Reward Models9 Oct
  4. 2023Statistical Rejection Sampling Improves Preference Optimization13 Sep
  5. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 22

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  2. 2022Generative Adversarial Networkscited by 5,603
  3. 2017Deep reinforcement learning from human preferencescited by 3,846
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  5. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  6. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  7. 2020Learning to summarize from human feedbackcited by 2,562
  8. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  9. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  10. 2016Concrete Problems in AI Safetycited by 1,582
  11. 2023PaLM 2 Technical Reportcited by 1,154
  12. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
10 more
  1. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  2. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  3. 2024Self-Rewarding Language Modelscited by 654
  4. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
  5. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 445
  6. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
  7. 2023A Long Way to Go: Investigating Length Correlations in RLHFcited by 285
  8. 2024A Minimaximalist Approach to Reinforcement Learning from Human Feedbackcited by 141
  9. 2024West-of-N: Synthetic Preferences for Self-Improving Reward Modelscited by 27
  10. 2022The Challenges of Exploration for Offline Reinforcement Learningcited by 26

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 91; 2025: 119; 2026: 47.

257 papers.

Sort:

By area:

  1. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
  2. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 451
  3. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 369
  4. 2024RLHF Workflow: From Reward Modeling to Online RLHFcited by 265
  5. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 215
  6. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergycited by 204
  7. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 184
  8. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferencescited by 176
  9. 2024Generative Reward Modelscited by 148
  10. 2024RLAIF-V: Open-Source AI Feedback Leads to Super GPT-4V Trustworthinesscited by 129
  11. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 129
  12. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
38 more
  1. 2024Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHFcited by 106
  2. 2024Improving Multi-Agent Debate with Sparse Communication Topologycited by 103
  3. 2024Learning Dynamics of LLM Finetuningcited by 93
  4. 2026Ministral 3cited by 88
  5. 2025Process Reward Models for LLM Agents: Practical Framework and Directionscited by 81
  6. 2024Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimizationcited by 78
  7. 2024RRM: Robust Reward Model Training Mitigates Reward Hackingcited by 75
  8. 2024Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHFcited by 73
  9. 2024BOND: Aligning LLMs with Best-of-N Distillationcited by 73
  10. 2024Systematic Evaluation of LLM-as-a-Judge in LLM Alignment Tasks: Explainable Metrics and Diverse Prompt Templatescited by 71
  11. 2024Robust Preference Optimization through Reward Model Distillationcited by 67
  12. 2024Aligning LLM Agents by Learning Latent Preference from User Editscited by 66
  13. 2024REBEL: Reinforcement Learning via Regressing Relative Rewardscited by 66
  14. 2024Building Math Agents with Multi-Turn Iterative Preference Learningcited by 66
  15. 2024Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Modelscited by 65
  16. 2024Self-Exploring Language Models: Active Preference Elicitation for Online Alignmentcited by 63
  17. 2025Tapered Off-Policy REINFORCE: Stable and efficient reinforcement learning for LLMscited by 61
  18. 2025All Roads Lead to Likelihood: The Value of Reinforcement Learning in Fine-Tuningcited by 58
  19. 2024Active Preference Learning for Large Language Modelscited by 55
  20. 2024JetMoE: Reaching Llama2 Performance with 0.1M Dollarscited by 55
  21. 2024The Importance of Online Data: Understanding Preference Fine-tuning via Coveragecited by 55
  22. 2024AgentOhana: Design Unified Data and Training Pipeline for Effective Agent Learningcited by 53
  23. 2024Learning to Clarify: Multi-turn Conversations with Action-Based Contrastive Self-Trainingcited by 53
  24. 2024Bootstrapping Language Models with DPO Implicit Rewardscited by 53
  25. 2025Reinforcement Learning from Human Feedbackcited by 53
  26. 2024Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignmentcited by 50
  27. 2024Offline Regularised Reinforcement Learning for Large Language Models Alignmentcited by 48
  28. 2024From Lists to Emojis: How Format Bias Affects Model Alignmentcited by 48
  29. 2024Rethinking Bradley-Terry Models in Preference-Based Reward Modeling: Foundations, Theory, and Alternativescited by 48
  30. 2025The Best Instruction-Tuning Data are Those That Fitcited by 48
  31. 2024Margin-aware Preference Optimization for Aligning Diffusion Models without Referencecited by 45
  32. 2025Voxtralcited by 41
  33. 2024A Survey on Data Synthesis and Augmentation for Large Language Modelscited by 40
  34. 2024Online DPO: Online Direct Preference Optimization with Fast-Slow Chasingcited by 38
  35. 2024Discovering Preference Optimization Algorithms with and for Large Language Modelscited by 38
  36. 2025NFT: Bridging Supervised Learning and Reinforcement Learning in Math Reasoningcited by 38
  37. 2024CREAM: Consistency Regularized Self-Rewarding Language Modelscited by 35
  38. 2025A Survey of Direct Preference Optimizationcited by 35

50 most cited of 257 All 257 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026From RLHF to Direct Alignment: A Theoretical Unification of Preference Learning for Large Language Models
  2. 2024BPO: Staying Close to the Behavior LLM Creates Better Online LLM Alignment
  3. 2025RLTHF: Targeted Human Feedback for LLM Alignment
  4. 2025A Principled Loss Function for Direct Language Model Alignment
  5. 2024OPTune: Efficient Online Preference Tuning
  6. 2025RLHF: A comprehensive Survey for Cultural, Multimodal and Low Latency Alignment Methods
  7. 2024Unified Preference Optimization: Language Model Alignment Beyond the Preference Frontier
  8. 2024Optimizing LLMs with Direct Preferences: A Data Efficiency Perspective