mlsanityData to 7 Oct 2026

SAIL: Self-Improving Efficient Online Alignment of Large Language Models

arXiv 2406.15567 ↗First posted 21 Jun 2024cs.LG cs.AI cs.CLcited by 20

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

also in 1 more
Abstract (arXiv)

Reinforcement Learning from Human Feedback (RLHF) is a key method for aligning large language models (LLMs) with human preferences. However, current offline alignment approaches like DPO, IPO, and SLiC rely heavily on fixed preference datasets, which can lead to sub-optimal performance. On the other hand, recent literature has focused on designing online RLHF methods but still lacks a unified conceptual formulation and suffers from distribution shift issues. To address this, we establish that online LLM alignment is underpinned by bilevel optimization. By reducing this formulation to an efficient single-level first-order method (using the reward-policy equivalence), our approach generates new samples and iteratively refines model alignment by exploring responses and regulating preference labels. In doing so, we permit alignment methods to operate in an online and self-improving manner, as well as generalize prior online RLHF methods as special cases. Compared to state-of-the-art iterative RLHF methods, our approach significantly improves alignment performance on open-sourced datasets with minimal computational overhead.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 27 of its 41 references that are in the map. First posted on arXiv 21 Jun 2024.

Builds on

27 of 41 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Understanding the performance gap between online and offline alignment algorithms14 May
  2. 2024Direct Language Model Alignment from Online AI Feedback7 Feb
  3. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 24

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  6. 2017Deep reinforcement learning from human preferencescited by 3,846
  7. 2023Qwen Technical Reportcited by 3,273
  8. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  9. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  10. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  11. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  12. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
12 more
  1. 2024Self-Rewarding Language Modelscited by 654
  2. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  3. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 298
  4. 2024Reinforcement learningcited by 236
  5. 2024Self-Play Preference Optimization for Language Model Alignmentcited by 228
  6. 2020Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detectioncited by 217
  7. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferencescited by 176
  8. 2019On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shiftcited by 174
  9. 2024A Minimaximalist Approach to Reinforcement Learning from Human Feedbackcited by 141
  10. 2022SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learningcited by 96
  11. 2024Principled Penalty-based Methods for Bilevel Reinforcement Learning and RLHFcited by 41
  12. 2024A Critical Evaluation of AI Feedback for Aligning Large Language Modelscited by 36

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 11; 2026: 5.

20 papers.

Sort:

By area:

  1. 2025Evolving Language Models without Labels: Majority Drives Selection, Novelty Promotes Variationcited by 44
  2. 2024Preference Optimization for Reasoning with Pseudo Feedbackcited by 38
  3. 2024IterComp: Iterative Composition-Aware Feedback Learning from Model Gallery for Text-to-Image Generationcited by 32
  4. 2024The Crucial Role of Samplers in Online Direct Preference Optimizationcited by 26
  5. 2025Extragradient Preference Optimization (EGPO): Beyond Last-Iterate Convergence for Nash Learning from Human Feedbackcited by 15
  6. 2025On The Sample Complexity Bounds In Bilevel Reinforcement Learningcited by 10
  7. 2025Understanding the Performance Gap in Preference Learning: A Dichotomy of RLHF and DPOcited by 10
  8. 2025Stop Reducing Responsibility in LLM-Powered Multi-Agent Systems to Local Alignmentcited by 7
  9. 2025Theoretical Modeling of Large Language Model Self-Improvement Training Dynamics Through Solver-Verifier Gapcited by 6
  10. 2025Toward Evaluative Thinking: Meta Policy Optimization with Evolving Reward Modelscited by 4
  11. 2024On The Global Convergence Of Online RLHF With Neural Parametrizationcited by 3
  12. 2025Bi-Level Policy Optimization with Nystr"om Hypergradientscited by 3
8 more
  1. 2025On the Role of Preference Variance in Preference Optimizationcited by 3
  2. 2026Explaining and Preventing Alignment Collapse in Iterative RLHFcited by 3
  3. 2025Lightweight Robust Direct Preference Optimizationcited by 2
  4. 2025Mitigating Reward Over-optimization in Direct Alignment Algorithms with Importance Samplingcited by 1
  5. 2026Oracle-Robust Online Alignment for Large Language Modelscited by 1
  6. 2026On the Convergence of Self-Improving Online LLM Alignmentcited by 1
  7. 2026IRIS: Interpolative R'enyi Iterative Self-play for Large Language Model Fine-Tuningcited by 0
  8. 2026Distributionally Robust Listwise Preference Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Online Preference Alignment for Language Models via Count-based Exploration
  2. 2024Sample-Efficient Alignment for LLMs
  3. 2026From RLHF to Direct Alignment: A Theoretical Unification of Preference Learning for Large Language Models
  4. 2025Beyond RLHF: A Unified Theoretical Framework of Alignment
  5. 2024Self-Exploring Language Models: Active Preference Elicitation for Online Alignment
  6. 2024Self-Play with Adversarial Critic: Provable and Scalable Offline Alignment for Language Models
  7. 2024Unified Preference Optimization: Language Model Alignment Beyond the Preference Frontier
  8. 2024Investigating on RLHF methodology