mlsanityData to 14 Oct 2026

Preference-based Online Learning with Dueling Bandits: A Survey

arXiv 1807.11398 ↗First posted 30 Jul 2018cs.LG stat.MLcited by 102

Not on the map. Why?

Cited by 36 in LLM alignment / RLHF / preference optimisation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In machine learning, the notion of multi-armed bandits refers to a class of online learning problems, in which an agent is supposed to simultaneously explore and exploit a given set of choice alternatives in the course of a sequential decision process. In the standard setting, the agent learns from stochastic feedback in the form of real-valued rewards. In many applications, however, numerical reward signals are not readily available -- instead, only weaker information is provided, in particular relative preferences in the form of qualitative comparisons between pairs of alternatives. This observation has motivated the study of variants of the multi-armed bandit problem, in which more general representations are used both for the type of feedback to learn from and the target of prediction. The aim of this paper is to provide a survey of the state of the art in this field, referred to as preference-based multi-armed bandits or dueling bandits. To this end, we provide an overview of problems that have been considered in the literature as well as methods for tackling them. Our taxonomy is mainly based on the assumptions made by these methods about the data-generating process and, related to this, the properties of the preference-based feedback.

Key facts. Cited by 102 papers in the map (data through 14 Oct 2026). First posted on arXiv 30 Jul 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 4; 2022: 13; 2023: 19; 2024: 21; 2025: 24; 2026: 21.

102 papers.

Sort:

By area:

  1. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 350
  2. 2024A Minimaximalist Approach to Reinforcement Learning from Human Feedbackcited by 144
  3. 2024DPO Meets PPO: Reinforced Token Optimization for RLHFcited by 127
  4. 2024Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizercited by 109
  5. 2023Is RLHF More Difficult than Standard RL?cited by 96
  6. 2022Human-in-the-loop: Provably Efficient Preference-based Reinforcement Learning with General Function Approximationcited by 81
  7. 2024Online Iterative Reinforcement Learning from Human Feedback with General Preference Modelcited by 52
  8. 2021Efficient and Optimal Algorithms for Contextual Dueling Bandits under Realizabilitycited by 44
  9. 2023PARL: A Unified Framework for Policy Alignment in Reinforcement Learning from Human Feedbackcited by 40
  10. 2022Stochastic Contextual Dueling Bandits under Linear Stochastic Transitivity Modelscited by 37
  11. 2023Zeroth-Order Optimization Meets Human Feedback: Provable Learning via Ranking Oraclescited by 35
  12. 2022A Survey of Methods for Automated Algorithm Configurationcited by 30
38 more
  1. 2023qEUBO: A Decision-Theoretic Acquisition Function for Preferential Bayesian Optimizationcited by 23
  2. 2023Variance-Aware Regret Bounds for Stochastic Contextual Dueling Banditscited by 22
  3. 2025Logarithmic Regret for Online KL-Regularized Reinforcement Learningcited by 22
  4. 2023Sample Efficient Preference Alignment in LLMs via Active Explorationcited by 18
  5. 2023Borda Regret Minimization for Generalized Linear Dueling Banditscited by 17
  6. 2024Optimal Design for Human Preference Elicitationcited by 17
  7. 2023Linear Partial Monitoring for Sequential Decision-Making: Algorithms, Regret Bounds and Applicationscited by 12
  8. 2023Differentially Private Reward Estimation with Preference Feedbackcited by 12
  9. 2024Zeroth-Order Policy Gradient for Reinforcement Learning from Human Feedback without Reward Inferencecited by 12
  10. 2021Bias-Robust Bayesian Optimization via Dueling Banditscited by 11
  11. 2022Active Evaluation: Efficient NLG Evaluation with Few Pairwise Comparisonscited by 10
  12. 2025ActiveDPO: Active Direct Preference Optimization for Sample-Efficient Alignmentcited by 10
  13. 2021Optimal and Efficient Dynamic Regret Algorithms for Non-Stationary Dueling Banditscited by 9
  14. 2022Diversified Recommendations for Agents with Adaptive Preferencescited by 9
  15. 2025VARP: Reinforcement Learning from Vision-Language Model Feedback with Agent Regularized Preferencescited by 9
  16. 2022ANACONDA: An Improved Dynamic Regret Algorithm for Adaptive Non-Stationary Dueling Banditscited by 8
  17. 2025Avoiding mathbfexp(Rmax) scaling in RLHF through Preference-based Explorationcited by 8
  18. 2025PABBO: Preferential Amortized Black-Box Optimizationcited by 8
  19. 2025LLM Prompt Duel Optimizer: Efficient Label-Free Prompt Optimizationcited by 8
  20. 2022Non-Stationary Dueling Banditscited by 7
  21. 2023Eliciting User Preferences for Personalized Multi-Objective Decision Making through Comparative Feedbackcited by 7
  22. 2023Near-Optimal Pure Exploration in Matrix Games: A Generalization of Stochastic Bandits & Dueling Banditscited by 7
  23. 2024Bandits with Preference Feedback: A Stackelberg Game Perspectivecited by 7
  24. 2024Neural Dueling Bandits: Preference-Based Optimization with Human Feedbackcited by 7
  25. 2025Addressing Concept Mislabeling in Concept Bottleneck Models Through Preference Optimizationcited by 7
  26. 2022Dueling Convex Optimization with General Preferencescited by 6
  27. 2022AC-Band: A Combinatorial Bandit-Based Approach to Algorithm Configurationcited by 6
  28. 2023Active Ranking of Experts Based on their Performances in Many Taskscited by 6
  29. 2024Active Preference Learning for Ordering Items In- and Out-of-samplecited by 6
  30. 2024Online Bandit Learning with Offline Preference Data for Improved RLHFcited by 6
  31. 2025A Statistical Case Against Empirical Human-AI Alignmentcited by 6
  32. 2026Towards a Theoretical Understanding to the Generalization of RLHFcited by 6
  33. 2022Finding Optimal Arms in Non-stochastic Combinatorial Bandits with Semi-bandit Feedback and Finite Budgetcited by 5
  34. 2022Versatile Dueling Bandits: Best-of-both-World Analyses for Online Learning from Preferencescited by 5
  35. 2024Conversational Dueling Bandits in Generalized Linear Modelscited by 5
  36. 2025Best Policy Learning from Trajectory Preference Feedbackcited by 5
  37. 2026Provably Efficient Regularized Online RLHF with Generalized Bilinear Preferencescited by 5
  38. 2023Kernelized Offline Contextual Dueling Banditscited by 4

50 most cited of 102 All 102 (JSON)