mlsanityData to 7 Oct 2026

PILAF: Optimal Human Preference Sampling for Reward Modeling

arXiv 2502.04270 ↗First posted 6 Feb 2025cs.LG stat.MLcited by 18

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

As large language models increasingly drive real-world applications, aligning them with human values becomes paramount. Reinforcement Learning from Human Feedback (RLHF) has emerged as a key technique, translating preference data into reward models when oracle human values remain inaccessible. In practice, RLHF mostly relies on approximate reward models, which may not consistently guide the policy toward maximizing the underlying human values. We propose Policy-Interpolated Learning for Aligned Feedback (PILAF), a novel response sampling strategy for preference labeling that explicitly aligns preference learning with maximizing the underlying oracle reward. PILAF is theoretically grounded, demonstrating optimality from both an optimization and a statistical perspective. The method is straightforward to implement and demonstrates strong performance in iterative and online RLHF settings where feedback curation is critical.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 21 of its 42 references that are in the map. First posted on arXiv 6 Feb 2025.

Builds on

21 of 42 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs24 Oct
  2. 2024Optimal Design for Reward Modeling in RLHF22 Oct
  3. 2024Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHF31 May
  4. 2024Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHF29 May
  5. 2024OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework20 May
  6. 2024Reinforcement Learning from Human Feedback with Active Queries14 Feb
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr

Other references · 14

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2023Let's Verify Step by Stepcited by 3,598
  6. 2024Direct Language Model Alignment from Online AI Feedbackcited by 257
  7. 2024KTO: Model Alignment as Prospect Theoretic Optimizationcited by 256
  8. 2023Some things are more CRINGE than others: Iterative Preference Optimization with the Pairwise Cringe Losscited by 108
  9. 2024Decoding-time Realignment of Language Modelscited by 67
  10. 2024Inference Scaling for Long-Context Retrieval Augmented Generationcited by 65
  11. 2024Self-Exploring Language Models: Active Preference Elicitation for Online Alignmentcited by 63
  12. 2024The Crucial Role of Samplers in Online Direct Preference Optimizationcited by 26
2 more
  1. 2023Sample Efficient Preference Alignment in LLMs via Active Explorationcited by 18
  2. 2024Sample-Efficient Alignment for LLMscited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 14; 2026: 4.

18 papers.

Sort:

By area:

  1. 2025Asymmetric REINFORCE for off-Policy Reinforcement Learning: Balancing positive and negative rewardscited by 29
  2. 2025Reinforce-Ada: An Adaptive Sampling Framework under Non-linear RL Objectivescited by 23
  3. 2026Self-Hinting Language Models Enhance Reinforcement Learningcited by 19
  4. 2025Extragradient Preference Optimization (EGPO): Beyond Last-Iterate Convergence for Nash Learning from Human Feedbackcited by 15
  5. 2025Don't Waste Mistakes: Leveraging Negative RL-Groups via Confidence Reweightingcited by 11
  6. 2025Rethinking Diverse Human Preference Learning through Principal Component Analysiscited by 10
  7. 2025Understanding the Performance Gap in Preference Learning: A Dichotomy of RLHF and DPOcited by 10
  8. 2025What Matters in Data for DPO?cited by 10
  9. 2025Inverse Reinforcement Learning Meets Large Language Model Post-Training: Basics, Advances, and Opportunitiescited by 7
  10. 2025Do Large Language Models (Really) Need Statistical Foundations?cited by 5
  11. 2025Limited Preference Data? Learning Better Reward Model with Latent Space Synthesiscited by 2
  12. 2025On the optimization dynamics of RLVR: Gradient gap and step size thresholdscited by 2
6 more
  1. 2026How Sampling Shapes LLM Alignment: From One-Shot Optima to Iterative Dynamicscited by 2
  2. 2026f-Divergence Regularized RLHF: Two Tales of Sampling and Unified Analysescited by 2
  3. 2025Understanding the Impact of Sampling Quality in Direct Preference Optimizationcited by 1
  4. 2025When Distance Distracts: Representation Distance Bias in BT-Loss for Reward Modelscited by 1
  5. 2026Reward Modeling for Multi-Agent Orchestrationcited by 1
  6. 2025Towards Efficient Online Exploration for Reinforcement Learning with Human Feedbackcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  2. 2024Towards Comprehensive Preference Data Collection for Reward Modeling
  3. 2025Influencing Humans to Conform to Preference Models for RLHF
  4. 2023Contrastive Preference Learning: Learning from Human Feedback without RL
  5. 2025Policy-labeled Preference Learning: Is Preference Enough for RLHF?
  6. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint
  7. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning
  8. 2025Capturing Individual Human Preferences with Reward Features