mlsanityData to 14 Oct 2026

Bootstrapping LLMs via Preference-Based Policy Optimization

arXiv 2511.12867 ↗First posted 17 Nov 2025cs.AI cs.LGcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

also in 1 more
Abstract (arXiv)

Bootstrapping large language models (LLMs) through preference-based policy optimization offers a promising direction for aligning model behavior with human preferences without relying on extensive manual annotations. In this work, we propose a novel preference-based policy optimization (PbPO) framework that formulates the learning process as a min-max game between the main policy and a reward model (RM). The RM is constrained within a confidence set derived from preference data to ensure reliable exploitation. Our iterative online algorithm actively collects preference data through guided exploration of the evolving policy, enabling continual self-improvement of both the policy and the RM. We provide theoretical guarantees for our method, establishing high-probability regret bounds for both settings with sequence-level RM and token-level RM, demonstrating its effectiveness in bootstrapping LLMs. Extensive experiments on five benchmarks show that our approach consistently outperforms existing state-of-the-art preference optimization techniques.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 32 of its 53 references that are in the map. First posted on arXiv 17 Nov 2025.

Builds on

32 of 53 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Faster WIND: Accelerating Iterative Best-of-N Distillation for LLM Alignment28 Oct
  2. 2024Bootstrapping Language Models with DPO Implicit Rewards14 Jun
  3. 2024Self-Exploring Language Models: Active Preference Elicitation for Online Alignment29 May
  4. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  5. 2024Direct Language Model Alignment from Online AI Feedback7 Feb
  6. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  7. 2023Leftover Lunch: Advantage-based Offline Reinforcement Learning for Language Models24 May
  8. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 24

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  7. 2017Deep reinforcement learning from human preferencescited by 3,912
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  9. 2020Learning to summarize from human feedbackcited by 2,588
  10. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  11. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  12. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
12 more
  1. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  2. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 716
  3. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 406
  4. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 350
  5. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisonscited by 247
  6. 2024WARM: On the Benefits of Weight Averaged Reward Modelscited by 147
  7. 2024DPO Meets PPO: Reinforced Token Optimization for RLHFcited by 127
  8. 2024BOND: Aligning LLMs with Best-of-N Distillationcited by 76
  9. 2024Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHFcited by 73
  10. 2024Active Preference Optimization for Sample Efficient RLHFcited by 62
  11. 2024Online Iterative Reinforcement Learning from Human Feedback with General Preference Modelcited by 52
  12. 2022On the Sensitivity of Reward Inference to Misspecified Human Modelscited by 25

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Optimizing LLMs with Direct Preferences: A Data Efficiency Perspective
  2. 2024ICPL: Few-shot In-context Preference Learning via LLMs
  3. 2024Unified Preference Optimization: Language Model Alignment Beyond the Preference Frontier
  4. 2025Offline Preference Optimization via Maximum Marginal Likelihood Estimation
  5. 2025Finding the Sweet Spot: Preference Data Construction for Scaling Preference Optimization
  6. 2024Multi-Reference Preference Optimization for Large Language Models
  7. 2025Mutual-Taught for Co-adapting Policy and Reward Models
  8. 2025A Survey of Direct Preference Optimization