mlsanityData to 14 Oct 2026

Offline Clustering of Preference Learning with Active-data Augmentation

arXiv 2510.26301 ↗First posted 30 Oct 2025cs.LGcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Preference learning from pairwise feedback is a widely adopted framework in applications such as reinforcement learning with human feedback and recommendations. In many practical settings, however, user interactions are limited or costly, making offline preference learning necessary. Moreover, real-world preference learning often involves users with different preferences. For example, annotators from different backgrounds may rank the same responses differently. This setting presents two central challenges: (1) identifying similarity across users to effectively aggregate data, especially under scenarios where offline data is imbalanced across dimensions, and (2) handling the imbalanced offline data where some preference dimensions are underrepresented. To address these challenges, we study the Offline Clustering of Preference Learning problem, where the learner has access to fixed datasets from multiple users with potentially different preferences and aims to maximize utility for a test user. To tackle the first challenge, we first propose Off-C2PL for the pure offline setting, where the learner relies solely on offline data. Our theoretical analysis provides a suboptimality bound that explicitly captures the tradeoff between sample noise and bias. To address the second challenge of inbalanced data, we extend our framework to the setting with active-data augmentation where the learner is allowed to select a limited number of additional active-data for the test user based on the cluster structure learned by Off-C2PL. In this setting, our second algorithm, A2-Off-C2PL, actively selects samples that target the least-informative dimensions of the test user's preference. We prove that these actively collected samples contribute more effectively than offline ones. Finally, we validate our theoretical results through simulations on synthetic and real-world datasets.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 49 of its 83 references that are in the map. First posted on arXiv 30 Oct 2025.

Builds on

49 of 83 references.

Closest ancestors · 12

Same direction, newest first

  1. 2024Dual Active Learning for Reinforcement Learning from Human Feedback3 Oct
  2. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning19 Aug
  3. 2024RLHF from Heterogeneous Feedback via Personalization and Preference Aggregation30 Apr
  4. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferences4 Apr
  5. 2024Active Preference Optimization for Sample Efficient RLHF16 Feb
  6. 2024Reinforcement Learning from Human Feedback with Active Queries14 Feb
  7. 2024Online Iterative Reinforcement Learning from Human Feedback with General Preference Model11 Feb
  8. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint18 Dec
  9. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback27 Jul
  10. 2023Reinforcement Learning with Human Feedback: Learning Dynamic Choices via Pessimism29 May
  11. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons26 Jan
  12. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr

Other references · 37

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2020Learning to summarize from human feedbackcited by 2,588
  4. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  5. 2019A Survey of Autonomous Driving: Common Practices and Emerging Technologiescited by 379
  6. 2020Is Pessimism Provably Efficient for Offline RL?cited by 364
  7. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 284
  8. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Mergingcited by 280
  9. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 279
  10. 2021Bellman-consistent Pessimism for Offline Reinforcement Learningcited by 269
  11. 2024Personalized Language Modeling from Personalized Human Feedbackcited by 132
  12. 2024BoNBoN Alignment for Large Language Models and the Sweetness of Best-of-n Samplingcited by 121
25 more
  1. 2014Online Clustering of Banditscited by 102
  2. 2018Preference-based Online Learning with Dueling Bandits: A Surveycited by 102
  3. 2015Contextual Dueling Banditscited by 101
  4. 2024Group Robust Preference Optimization in Reward-free RLHFcited by 95
  5. 2023Personalisation within bounds: A risk taxonomy and policy framework for the alignment of large language models with personalised feedbackcited by 87
  6. 2022Human-in-the-loop: Provably Efficient Preference-based Reinforcement Learning with General Function Approximationcited by 81
  7. 2024On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularizationcited by 72
  8. 2020Preference-based Reinforcement Learning with Finite-Time Guaranteescited by 65
  9. 2021Efficient and Optimal Algorithms for Contextual Dueling Bandits under Realizabilitycited by 44
  10. 2019Improved Algorithm on Online Clustering of Banditscited by 38
  11. 2024Stepwise Alignment for Constrained Language Model Policy Optimizationcited by 30
  12. 2018Dynamic Assortment Optimization with Changing Contextual Informationcited by 28
  13. 2021On the Optimality of Batch Policy Optimization Algorithmscited by 28
  14. 2024Spread Preference Annotation: Direct Preference Judgment for Efficient LLM Alignmentcited by 27
  15. 2021Reinforcement Learning for Education: Opportunities and Challengescited by 25
  16. 2017Online Clustering of Contextual Cascading Banditscited by 24
  17. 2019Warm-starting Contextual Bandits: Robustly Combining Supervised and Bandit Feedbackcited by 21
  18. 2024Nearly Minimax Optimal Regret for Multinomial Logistic Banditcited by 20
  19. 2023Oracle-Efficient Pessimism: Offline Policy Optimization in Contextual Banditscited by 15
  20. 2019Online Pricing with Offline Data: Phase Transition and Inverse Square Lawcited by 13
  21. 2022Artificial Replay: A Meta-Algorithm for Harnessing Historical Data in Banditscited by 13
  22. 2023Online Clustering of Bandits with Misspecified User Modelscited by 12
  23. 2024Online Bandits with (Biased) Offline Data: Adaptive Learning under Distribution Mismatchcited by 10
  24. 2022Near-optimal Policy Identification in Active Reinforcement Learningcited by 7
  25. 2023Online Corrupted User Detection and Regret Minimizationcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Benchmarks and Algorithms for Offline Preference-Based Reward Learning
  2. 2026From Reward-Free Representations to Preferences: Rethinking Offline Preference-Based Reinforcement Learning
  3. 2023Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF
  4. 2024Preference Elicitation for Offline Reinforcement Learning
  5. 2023Sample Efficient Preference Alignment in LLMs via Active Exploration
  6. 2024Direct Preference Optimization With Unobserved Preference Heterogeneity: The Necessity of Ternary Preferences
  7. 2025Direct Preference Optimization with Unobserved Preference Heterogeneity: The Necessity of Ternary Preferences
  8. 2024PAL: Pluralistic Alignment Framework for Learning from Heterogeneous Preferences