mlsanityData to 14 Oct 2026

Direct Advantage Regression: Aligning LLMs with Online AI Reward

arXiv 2504.14177 ↗First posted 19 Apr 2025cs.AI cs.CL cs.HCcited by 1

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Online AI Feedback (OAIF) presents a promising alternative to Reinforcement Learning from Human Feedback (RLHF) by utilizing online AI preference in aligning language models (LLMs). However, the straightforward replacement of humans with AI deprives LLMs from learning more fine-grained AI supervision beyond binary signals. In this paper, we propose Direct Advantage Regression (DAR), a simple alignment algorithm using online AI reward to optimize policy improvement through weighted supervised fine-tuning. As an RL-free approach, DAR maintains theoretical consistency with online RLHF pipelines while significantly reducing implementation complexity and improving learning efficiency. Our empirical results underscore that AI reward is a better form of AI supervision consistently achieving higher human-AI agreement as opposed to AI preference. Additionally, evaluations using GPT-4-Turbo and MT-bench show that DAR outperforms both OAIF and online RLHF baselines.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 25 of its 40 references that are in the map. First posted on arXiv 19 Apr 2025.

Builds on

25 of 40 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Bootstrapping Language Models with DPO Implicit Rewards14 Jun
  2. 2024Online DPO: Online Direct Preference Optimization with Fast-Slow Chasing8 Jun
  3. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study16 Apr
  4. 2024Direct Language Model Alignment from Online AI Feedback7 Feb
  5. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  6. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructions20 Dec

Other references · 19

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  6. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  7. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  8. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
  9. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  10. 2022Large Language Models Can Self-Improvecited by 664
  11. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 625
  12. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compressioncited by 453
7 more
  1. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 406
  2. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 401
  3. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 350
  4. 2024Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMscited by 142
  5. 2024HelpSteer2-Preference: Complementing Ratings with Preferencescited by 130
  6. 2024Beyond the Limits: A Survey of Techniques to Extend the Context Length in Large Language Modelscited by 76
  7. 2024Cost-Effective Proxy Reward Model Construction with On-Policy and Active Learningcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026A Survey of Reinforcement Learning for Large Language Models under Data Scarcity: Challenges and Solutionscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs
  2. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints
  3. 2024Exploring RL-based LLM Training for Formal Language Tasks with Programmed Rewards
  4. 2026From RLHF to Direct Alignment: A Theoretical Unification of Preference Learning for Large Language Models
  5. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  6. 2024On The Global Convergence Of Online RLHF With Neural Parametrization
  7. 2023Leftover Lunch: Advantage-based Offline Reinforcement Learning for Language Models
  8. 2024Online Merging Optimizers for Boosting Rewards and Mitigating Tax in Alignment