mlsanityData to 7 Oct 2026

Optimal Design for Human Preference Elicitation

arXiv 2404.13895 ↗First posted 22 Apr 2024cs.LGcited by 17

Not on the map. Why?

Abstract (arXiv)

Learning of preference models from human feedback has been central to recent advances in artificial intelligence. Motivated by the cost of obtaining high-quality human annotations, we study efficient human preference elicitation for learning preference models. The key idea in our work is to generalize optimal designs, an approach to computing optimal information-gathering policies, to lists of items that represent potential questions with answers. The policy is a distribution over the lists and we elicit preferences from them proportionally to their probabilities. To show the generality of our ideas, we study both absolute and ranking feedback models on items in the list. We design efficient algorithms for both and analyze them. Finally, we demonstrate that our algorithms are practical by evaluating them on existing question-answering problems.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Builds on 56 of its 101 references that are in the map. First posted on arXiv 22 Apr 2024.

Builds on

56 of 101 references.

References · 56

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  3. 2017Deep reinforcement learning from human preferencescited by 3,846
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  5. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  6. 2016CVXPY: A Python-Embedded Modeling Language for Convex Optimizationcited by 633
  7. 2019Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Boundscited by 602
  8. 2020A Survey of Deep Active Learningcited by 547
  9. 2020Is Pessimism Provably Efficient for Offline RL?cited by 362
  10. 2022One Embedder, Any Task: Instruction-Finetuned Text Embeddingscited by 329
  11. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 275
  12. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisonscited by 244
38 more
  1. 2023Large Language Model Alignment: A Surveycited by 241
  2. 2015Doubly Robust Policy Evaluation and Optimizationcited by 158
  3. 2020Cold-start Active Learning through Self-supervised Language Modelingcited by 128
  4. 2013A Theoretical Analysis of NDCG Type Ranking Measurescited by 119
  5. 2020Reward-rational (implicit) choice: A unifying formalism for reward learningcited by 119
  6. 2021Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learningcited by 109
  7. 2021B-Pref: Benchmarking Preference-Based Reinforcement Learningcited by 106
  8. 2021Dueling RL: Reinforcement Learning with Trajectory Preferencescited by 105
  9. 2018Preference-based Online Learning with Dueling Bandits: A Surveycited by 100
  10. 2014Best-Arm Identification in Linear Banditscited by 98
  11. 2023Is RLHF More Difficult than Standard RL?cited by 96
  12. 2019Sequential Experimental Design for Transductive Linear Banditscited by 88
  13. 2021Gone Fishing: Neural Active Learning with Fisher Embeddingscited by 88
  14. 2022Human-in-the-loop: Provably Efficient Preference-based Reinforcement Learning with General Function Approximationcited by 80
  15. 2015Cascading Bandits: Learning to Rank in the Cascade Modelcited by 75
  16. 2020Active Preference-Based Gaussian Process Regression for Reward Learningcited by 70
  17. 2023Inverse Preference Learning: Preference-based RL without a Reward Functioncited by 68
  18. 2020Preference-based Reinforcement Learning with Finite-Time Guaranteescited by 65
  19. 2024Active Preference Optimization for Sample Efficient RLHFcited by 61
  20. 2019Dueling Posterior Sampling for Preference-Based Reinforcement Learningcited by 56
  21. 2023Making RL with Preference-based Feedback Efficient via Randomizationcited by 46
  22. 2016Cascading Bandits for Large-Scale Recommendation Problemscited by 45
  23. 2020An Empirical Process Approach to the Union Bound: Practical Algorithms for Combinatorial and Linear Banditscited by 45
  24. 2021Efficient and Optimal Algorithms for Contextual Dueling Bandits under Realizabilitycited by 44
  25. 2023Beyond Reward: Offline Preference-guided Policy Optimizationcited by 35
  26. 2021Minimax Optimal Fixed-Budget Best Arm Identification in Linear Banditscited by 32
  27. 2018TopRank: A practical algorithm for online stochastic rankingcited by 30
  28. 2016Multiple-Play Bandits in the Position-Based Modelcited by 27
  29. 2017Online Learning to Rank in Stochastic Click Modelscited by 26
  30. 2023Provable Reward-Agnostic Preference-Based Reinforcement Learningcited by 25
  31. 2021Fixed-Budget Best-Arm Identification in Structured Banditscited by 22
  32. 2023Towards Practical Preferential Bayesian Optimization with Skew Gaussian Processescited by 19
  33. 2024Principled Preferential Bayesian Optimizationcited by 16
  34. 2021Improved Algorithms for Agnostic Pool-based Active Classificationcited by 15
  35. 2021Pure Exploration in Kernel and Neural Banditscited by 13
  36. 2019Zeroth Order Non-convex optimization with Dueling-Choice Banditscited by 11
  37. 2020Chernoff Sampling for Active Testing and Extension to Active Regressioncited by 11
  38. 2021Bias-Robust Bayesian Optimization via Dueling Banditscited by 11

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 9; 2026: 6.

17 papers.

Sort:

By area:

  1. 2024Optimal Design for Reward Modeling in RLHFcited by 23
  2. 2025FisherSFT: Data-Efficient Supervised Fine-Tuning of Language Models Using Information Gaincited by 14
  3. 2025Robust Reinforcement Learning from Human Feedback for Large Language Models Fine-Tuningcited by 13
  4. 2025Active Learning for Direct Preference Optimizationcited by 9
  5. 2025A Statistical Case Against Empirical Human-AI Alignmentcited by 6
  6. 2024Comparing Few to Rank Many: Active Human Preference Learning using Randomized Frank-Wolfecited by 5
  7. 2026Reinforcement Learning from Human Feedback: A Statistical Perspectivecited by 4
  8. 2025MisoDICE: Multi-Agent Imitation from Unlabeled Mixed-Quality Demonstrationscited by 3
  9. 2025Towards Cognitively-Faithful Decision-Making Models to Improve AI Alignmentcited by 2
  10. 2025Preference-Guided Learning for Sparse-Reward Multi-Agent Reinforcement Learningcited by 2
  11. 2025Preference-based Reinforcement Learning beyond Pairwise Comparisons: Benefits of Multiple Optionscited by 2
  12. 2026MASS-DPO: Multi-negative Active Sample Selection for Direct Policy Optimizationcited by 1
5 more
  1. 2025Cold-Start Active Preference Learning in Socio-Economic Domainscited by 0
  2. 2026Nearly Optimal Active Preference Learning and Its Application to LLM Alignmentcited by 0
  3. 2026Optimal Design for Multinomial Logit Model with Applications to Best Assortment Identificationcited by 0
  4. 2026Population-Level Generative Modeling for Ranking Datacited by 0
  5. 2026Beyond Pairwise Feedback: Listwise Vision-Language Supervision for Preference-Based Reward Learningcited by 0