mlsanityData to 7 Oct 2026

Comparing Few to Rank Many: Active Human Preference Learning using Randomized Frank-Wolfe

arXiv 2412.19396 ↗First posted 27 Dec 2024cs.LG cs.AI cs.ITcited by 5

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

We study learning of human preferences from a limited comparison feedback. This task is ubiquitous in machine learning. Its applications such as reinforcement learning from human feedback, have been transformational. We formulate this problem as learning a Plackett-Luce model over a universe of N choices from K-way comparison feedback, where typically K ll N. Our solution is the D-optimal design for the Plackett-Luce objective. The design defines a data logging policy that elicits comparison feedback for a small collection of optimally chosen points from all N choose K feasible subsets. The main algorithmic challenge in this work is that even fast methods for solving D-optimal designs would have O(N choose K) time complexity. To address this issue, we propose a randomized Frank-Wolfe (FW) algorithm that solves the linear maximization sub-problems in the FW method on randomly chosen variables. We analyze the algorithm, and evaluate it empirically on synthetic and open-source NLP datasets.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 12 of its 60 references that are in the map. First posted on arXiv 27 Dec 2024.

Builds on

12 of 60 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Active Preference Optimization for Sample Efficient RLHF16 Feb
  2. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback27 Jul
  3. 2023Beyond Reward: Offline Preference-guided Policy Optimization25 May
  4. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons26 Jan

Other references · 8

  1. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  2. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,841
  3. 2023Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agentscited by 356
  4. 2023Sample Efficient Preference Alignment in LLMs via Active Explorationcited by 18
  5. 2024Optimal Design for Human Preference Elicitationcited by 17
  6. 2022Provably Efficient Offline Reinforcement Learning with Trajectory-Wise Rewardcited by 13
  7. 2024Embedding-Aligned Language Modelscited by 9
  8. 2023Solving the Optimal Experiment Design Problem with Mixed-Integer Convex Methodscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 1.

5 papers.

  1. 2025FisherSFT: Data-Efficient Supervised Fine-Tuning of Language Models Using Information Gaincited by 14
  2. 2025Active Learning for Direct Preference Optimizationcited by 9
  3. 2025Less is More: Improving LLM Alignment via Preference Data Selectioncited by 4
  4. 2025Preference-based Reinforcement Learning beyond Pairwise Comparisons: Benefits of Multiple Optionscited by 2
  5. 2026MASS-DPO: Multi-negative Active Sample Selection for Direct Policy Optimizationcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2023Making RL with Preference-based Feedback Efficient via Randomization
  2. 2025Bayesian Optimization from Human Feedback: Near-Optimal Regret Bounds
  3. 2024Direct Preference Optimization With Unobserved Preference Heterogeneity: The Necessity of Ternary Preferences
  4. 2025Direct Preference Optimization with Unobserved Preference Heterogeneity: The Necessity of Ternary Preferences
  5. 2024A Minimaximalist Approach to Reinforcement Learning from Human Feedback
  6. 2026Optimal Design for Active Preference Learning with Biased LLM Judges
  7. 2023A density estimation perspective on learning from pairwise human preferences
  8. 2024Reward Modeling with Ordinal Feedback: Wisdom of the Crowd