mlsanityData to 14 Oct 2026

Alignment through Meta-Weighted Online Sampling: Bridging the Gap between Data Generation and Preference Optimization

arXiv 2509.23371 ↗First posted 27 Sep 2025cs.CL cs.AI cs.LGcited by 3

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Preference optimization is crucial for aligning large language models (LLMs) with human values and intentions. A significant challenge in this process is the distribution mismatch between pre-collected offline preference data and the evolving model policy. Existing methods attempt to reduce this gap using static heuristics or decoupled online sampling strategies, but they often fail to adapt to the model's dynamic learning state. To bridge this gap, we propose Meta-Weighted Adaptive Preference Optimization (MetaAPO), a novel framework that dynamically couples data generation with model training. MetaAPO employs a lightweight meta-learner, as an "alignment gap estimator", to evaluate the potential benefits of on-policy sampling in relation to offline data. This guides targeted online generation and assigns sample-wise meta-weights to the optimization objective, dynamically balancing the quality and distribution of online and offline data. Experiments on AlpacaEval 2, Arena-Hard and MT-Bench demonstrate that MetaAPO consistently outperforms existing preference optimization approaches across various settings, while reducing 42% in online annotation costs. Code is available at https://github.com/junming-yang/MetaAPO.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 46 of its 59 references that are in the map. First posted on arXiv 27 Sep 2025.

Builds on

46 of 59 references.

Closest ancestors · 23

Same direction, newest first

  1. 2025Larger or Smaller Reward Margins to Select Preferences for Alignment?25 Feb
  2. 2025Principled Data Selection for Alignment: The Hidden Risks of Difficult Examples11 Feb
  3. 2025DPO-Shift: Shifting the Distribution of Direct Preference Optimization11 Feb
  4. 2025Reveal the Mystery of DPO: The Connection between DPO and RL Algorithms5 Feb
  5. 2024SeRA: Self-Reviewing and Alignment of Large Language Models using Implicit Reward Margins12 Oct
  6. 2024Reward-Augmented Data Enhances Direct Preference Alignment of LLMs10 Oct
  7. 2024Self-Boosting Large Language Models with Synthetic Preference Data9 Oct
  8. 2024Progressively Label Enhancement for Large Language Model Alignment5 Aug
  9. 2024Bootstrapping Language Models with DPO Implicit Rewards14 Jun
  10. 2024Discovering Preference Optimization Algorithms with and for Large Language Models12 Jun
  11. 20243D-Properties: Identifying Challenges in DPO and Charting a Path Forward11 Jun
  12. 2024Self-Exploring Language Models: Active Preference Elicitation for Online Alignment29 May
11 more
  1. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  2. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study16 Apr
  3. 2024Mixed Preference Optimization: Reinforcement Learning with Data Selection and Better Reference Model28 Mar
  4. 2024RS-DPO: A Hybrid Rejection Sampling and Direct Preference Optimization Method for Alignment of Large Language Models15 Feb
  5. 2024Active Preference Learning for Large Language Models12 Feb
  6. 2024Direct Language Model Alignment from Online AI Feedback7 Feb
  7. 2024Human-Instruction-Free LLM Self-Alignment with Limited Samples6 Jan
  8. 2023What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning25 Dec
  9. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  10. 2023Beyond One-Preference-Fits-All Alignment: Multi-Objective Direct Preference Optimization5 Oct
  11. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 23

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2024Qwen2.5 Technical Reportcited by 4,942
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  7. 2020Learning to summarize from human feedbackcited by 2,588
  8. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  9. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 686
  10. 2024Self-Rewarding Language Modelscited by 681
  11. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 532
  12. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 500
11 more
  1. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 463
  2. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 406
  3. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 350
  4. 2025LMM-R1: Empowering 3B LMMs with Strong Reasoning Abilities Through Two-Stage Rule-Based RLcited by 248
  5. 2024Self-Play Preference Optimization for Language Model Alignmentcited by 232
  6. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 185
  7. 2014Label Distribution Learningcited by 124
  8. 2024Active Preference Optimization for Sample Efficient RLHFcited by 62
  9. 2024Reinforcement Learning from Human Feedback with Active Queriescited by 43
  10. 2024Not All Preference Pairs Are Created Equal: A Recipe for Annotation-Efficient Iterative Preference Learningcited by 7
  11. 2025More is Less: The Pitfalls of Multi-Model Synthetic Preference Data in DPO Safety Alignmentcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026BAPO: Boundary-Aware Policy Optimization for Reliable Agentic Searchcited by 3
  2. 2026Smaller Models, Better Rejects: Preference Distillation Scalingcited by 1
  3. 2026VRM: Teaching Reward Models to Understand Authentic Human Preferencescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Towards Bridging the Gap Between Offline and Iterative Alignment via Preference Distillation
  2. 2024MetaAlign: Align Large Language Models with Diverse Preferences during Inference Time
  3. 2026Meta-Aligner: Bidirectional Preference-Policy Optimization for Multi-Objective LLMs Alignment
  4. 2025Finding the Sweet Spot: Preference Data Construction for Scaling Preference Optimization
  5. 2024Towards a Unified View of Preference Learning for Large Language Models: A Survey
  6. 2024Preference-Guided Reflective Sampling for Aligning Language Models
  7. 2026How Sampling Shapes LLM Alignment: From One-Shot Optima to Iterative Dynamics
  8. 2026Meta-Learning Preferences for Multilingual LLM Alignment