mlsanityData to 14 Oct 2026

BPO: Staying Close to the Behavior LLM Creates Better Online LLM Alignment

arXiv 2406.12168 ↗First posted 18 Jun 2024cs.LG cs.AI cs.CLcited by 9

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Direct alignment from preferences (DAP) has emerged as a promising paradigm for aligning large language models (LLMs) to human desiderata from pre-collected, offline preference datasets. While recent studies indicate that existing offline DAP methods can directly benefit from online training samples, we highlight the need to develop specific online DAP algorithms to fully harness the power of online training. Specifically, we identify that the learned LLM should adhere to the proximity of the behavior LLM, which collects the training samples. To this end, we propose online Preference Optimization in proximity to the Behavior LLM (BPO), emphasizing the importance of constructing a proper trust region for LLM alignment. We conduct extensive experiments to validate the effectiveness and applicability of our approach by integrating it with various DAP methods, resulting in significant performance improvements across a wide range of tasks when training with the same amount of preference data. Even when only introducing one additional data collection phase, our online BPO improves its offline DAP baseline from 72.0% to 80.2% on TL;DR and from 82.2% to 89.1% on Anthropic Helpfulness in terms of win rate against human reference text.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 31 of its 40 references that are in the map. First posted on arXiv 18 Jun 2024.

Builds on

31 of 40 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Understanding the performance gap between online and offline alignment algorithms14 May
  2. 2024Learn Your Reference Model for Real Good Alignment15 Apr
  3. 2024Human Alignment of Large Language Models through Online Preference Optimisation13 Mar
  4. 2024Generalized Preference Optimization: A Unified Approach to Offline Alignment8 Feb
  5. 2023Some things are more CRINGE than others: Iterative Preference Optimization with the Pairwise Cringe Loss27 Dec
  6. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  7. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 24

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  7. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  8. 2017Deep reinforcement learning from human preferencescited by 3,912
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  10. 2015Continuous control with deep reinforcement learningcited by 3,407
  11. 2020Learning to summarize from human feedbackcited by 2,588
  12. 2015Trust Region Policy Optimizationcited by 2,580
12 more
  1. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  2. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  3. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  4. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  5. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  6. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
  7. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 401
  8. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 185
  9. 2019On the Weaknesses of Reinforcement Learning for Neural Machine Translationcited by 82
  10. 2021The Difficulty of Passive Learning in Deep Reinforcement Learningcited by 50
  11. 2022Offline Reinforcement Learning with Closed-Form Policy Improvement Operatorscited by 14
  12. 2022Your Autoregressive Generative Model Can be Better If You Treat It as an Energy-Based Onecited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 4.

9 papers.

  1. 2025Bridging Offline and Online Reinforcement Learning for LLMscited by 25
  2. 2025Robust Reinforcement Learning from Human Feedback for Large Language Models Fine-Tuningcited by 13
  3. 2025Doubly Robust Alignment for Large Language Modelscited by 12
  4. 2025LLM Alignment as Retriever Optimization: An Information Retrieval Perspectivecited by 7
  5. 2025Alleviating Distribution Shift in Synthetic Data for Machine Translation Quality Estimationcited by 4
  6. 2026Unifying Stable Optimization and Reference Regularization in RLHFcited by 4
  7. 2026How Sampling Shapes LLM Alignment: From One-Shot Optima to Iterative Dynamicscited by 2
  8. 2026Structuring the Space of Sociotechnical Alignmentcited by 2
  9. 2026Random Is Hard to Beat: Active Selection in online DPO with Modern LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Direct Language Model Alignment from Online AI Feedback
  2. 2026Towards Bridging the Gap Between Offline and Iterative Alignment via Preference Distillation
  3. 2025Less is More: Improving LLM Alignment via Preference Data Selection
  4. 2024Refined Direct Preference Optimization with Synthetic Data for Behavioral Alignment of LLMs
  5. 2025ActiveDPO: Active Direct Preference Optimization for Sample-Efficient Alignment
  6. 2024Optimizing LLMs with Direct Preferences: A Data Efficiency Perspective
  7. 2024Spread Preference Annotation: Direct Preference Judgment for Efficient LLM Alignment
  8. 2024REAL: Response Embedding-based Alignment for LLMs