mlsanityData to 14 Oct 2026

BPO: Towards Balanced Preference Optimization between Knowledge Breadth and Depth in Alignment

arXiv 2411.10914 ↗First posted 16 Nov 2024cs.CLcited by 20

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Reinforcement Learning with Human Feedback (RLHF) is the key to the success of large language models (LLMs) in recent years. In this work, we first introduce the concepts of knowledge breadth and knowledge depth, which measure the comprehensiveness and depth of an LLM or knowledge source respectively. We reveal that the imbalance in the number of prompts and responses can lead to a potential disparity in breadth and depth learning within alignment tuning datasets by showing that even a simple uniform method for balancing the number of instructions and responses can lead to significant improvements. Building on this, we further propose Balanced Preference Optimization (BPO), designed to dynamically augment the knowledge depth of each sample. BPO is motivated by the observation that the usefulness of knowledge varies across samples, necessitating tailored learning of knowledge depth. To achieve this, we introduce gradient-based clustering, estimating the knowledge informativeness and usefulness of each augmented sample based on the model's optimization direction. Our experimental results across various benchmarks demonstrate that BPO outperforms other baseline methods in alignment tuning while maintaining training efficiency. Furthermore, we conduct a detailed analysis of each component of BPO, providing guidelines for future research in preference data optimization.

Key facts. Cited by 20 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 30 of its 45 references that are in the map. First posted on arXiv 16 Nov 2024.

Builds on

30 of 45 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  2. 2024Insights into Alignment: Evaluating DPO and its Variants Across Multiple Tasks23 Apr
  3. 2024Scaling Data Diversity for Fine-Tuning Language Models in Human Alignment17 Mar
  4. 2024Curry-DPO: Enhancing Alignment using Curriculum Learning & Ranked Preferences12 Mar
  5. 2023What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning25 Dec
  6. 2023Statistical Rejection Sampling Improves Preference Optimization13 Sep
  7. 2023Fine-Tuning Language Models with Advantage-Induced Policy Alignment4 Jun
  8. 2023LIMA: Less Is More for Alignment18 May

Other references · 22

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  8. 2017Deep reinforcement learning from human preferencescited by 3,912
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  10. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  11. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 656
  12. 2020Estimating Training Data Influence by Tracing Gradient Descentcited by 580
10 more
  1. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 532
  2. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 500
  3. 2024A Survey on Data Selection for Language Modelscited by 236
  4. 2023Benchmarking Foundation Models with Language-Model-as-an-Examinercited by 172
  5. 2023Human-like Summarization Evaluation with ChatGPTcited by 126
  6. 2022Calibrating Sequence likelihood Improves Conditional Language Generationcited by 121
  7. 2023MM-LIMA: Less Is More for Alignment in Multi-Modal Datasetscited by 61
  8. 2024DALK: Dynamic Co-Augmentation of LLMs and KG to answer Alzheimer's Disease Questions with Scientific Literaturecited by 50
  9. 2023Understanding In-Context Learning via Supportive Pretraining Datacited by 41
  10. 2024Self-Training with Direct Preference Optimization Improves Chain-of-Thought Reasoningcited by 26

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 11; 2026: 7.

20 papers.

Sort:

By area:

  1. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 494
  2. 2025Preference Leakage: A Contamination Problem in LLM-as-a-judgecited by 125
  3. 2025TreeReview: A Dynamic Tree of Questions Framework for Deep and Efficient LLM-based Scientific Peer Reviewcited by 31
  4. 2026Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Modelscited by 27
  5. 2025Understanding and Mitigating Overrefusal in LLMs from an Unveiling Perspective of Safety Decision Boundarycited by 18
  6. 2025TL;DR: Too Long, Do Re-weighting for Efficient LLM Reasoning Compressioncited by 16
  7. 2025Find Your Optimal Teacher: Personalized Data Synthesis via Router-Guided Multi-Teacher Distillationcited by 15
  8. 2026ToolPRMBench: Evaluating and Advancing Process Reward Models for Tool-using Agentscited by 14
  9. 2025Larger or Smaller Reward Margins to Select Preferences for Alignment?cited by 11
  10. 2025The Quest for Efficient Reasoning: A Data-Centric Benchmark to CoT Distillationcited by 9
  11. 2025Who's Your Judge? On the Detectability of LLM-Generated Judgmentscited by 8
  12. 2025LLM Safety Alignment is Divergence Estimation in Disguisecited by 6
8 more
  1. 2025GuiLoMo: Allocating Expert Number and Rank for LoRA-MoE via Bilevel Optimization with GuidedSelection Vectorscited by 6
  2. 2026Diagnosis Before Recovery: Turning Agent Failures into Selective Self-Correctioncited by 3
  3. 2024Assessing the Impact of Conspiracy Theories Using Large Language Modelscited by 2
  4. 2025From Fragments to Facts: A Curriculum-Driven DPO Approach for Generating Hindi News Veracity Explanationscited by 2
  5. 2026How to Train a Real-World Silicon Concierge? Internalizing Complex Business Workflow to Only OneModelcited by 2
  6. 2026CURA: Clinical Uncertainty Risk Alignment for Language Model-Based Risk Predictioncited by 0
  7. 2026ARC: Fair Relative Advantage Comparison in Open-Ended Real-World Interactioncited by 0
  8. 2026Ask, Condition or Abstain: Reinforcement Learning for Missing-Premise Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025BPO: Revisiting Preference Modeling in Direct Preference Optimization
  2. 2025Gradient-Adaptive Policy Optimization: Towards Multi-Objective Alignment of Large Language Models
  3. 2025Leveraging Machine Unlearning for Cost-Efficient Preference Alignment
  4. 2024Unified Preference Optimization: Language Model Alignment Beyond the Preference Frontier
  5. 2026From RLHF to Direct Alignment: A Theoretical Unification of Preference Learning for Large Language Models
  6. 2024Offline Regularised Reinforcement Learning for Large Language Models Alignment
  7. 2025PEO: Improving Bi-Factorial Preference Alignment with Post-Training Policy Extrapolation
  8. 2025Beyond RLHF: A Unified Theoretical Framework of Alignment