mlsanityData to 7 Oct 2026

Neural Dueling Bandits: Preference-Based Optimization with Human Feedback

arXiv 2407.17112 ↗First posted 24 Jul 2024cs.LG cs.AI stat.MLcited by 7

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Contextual dueling bandit is used to model the bandit problems, where a learner's goal is to find the best arm for a given context using observed noisy human preference feedback over the selected arms for the past contexts. However, existing algorithms assume the reward function is linear, which can be complex and non-linear in many real-life applications like online recommendations or ranking web search results. To overcome this challenge, we use a neural network to estimate the reward function using preference feedback for the previously selected arms. We propose upper confidence bound- and Thompson sampling-based algorithms with sub-linear regret guarantees that efficiently select arms in each round. We also extend our theoretical results to contextual bandit problems with binary feedback, which is in itself a non-trivial contribution. Experimental results on the problem instances derived from synthetic datasets corroborate our theoretical results.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 28 of its 55 references that are in the map. First posted on arXiv 24 Jul 2024.

Builds on

28 of 55 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs12 Apr
  2. 2024Reinforcement Learning from Human Feedback with Active Queries14 Feb
  3. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons26 Jan
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr

Other references · 24

  1. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  3. 2019Introduction to Multi-Armed Banditscited by 475
  4. 2017On Kernelized Multi-armed Banditscited by 348
  5. 2022Teaching language models to support answers with verified quotescited by 290
  6. 2013Finite-Time Analysis of Kernelised Contextual Banditscited by 177
  7. 2020Neural Thompson Samplingcited by 127
  8. 2020Improved Optimistic Algorithms for Logistic Banditscited by 117
  9. 2018Preference-based Online Learning with Dueling Bandits: A Surveycited by 100
  10. 2021Feel-Good Thompson Sampling for Contextual Bandits and Reinforcement Learningcited by 65
  11. 2017Scalable Generalized Linear Bandits: Online Computation and Hashingcited by 63
  12. 2014Reducing Dueling Bandits to Cardinal Banditscited by 62
12 more
  1. 2015Regret Lower Bound and Optimal Algorithm in Dueling Bandit Problemcited by 51
  2. 2021Efficient and Optimal Algorithms for Contextual Dueling Bandits under Realizabilitycited by 44
  3. 2021Neural Contextual Bandits without Regretcited by 40
  4. 2022Stochastic Contextual Dueling Bandits under Linear Stochastic Transitivity Modelscited by 35
  5. 2024Prompt Optimization with Human Feedbackcited by 35
  6. 2016A Relative Exponential Weighing Algorithm for Adversarial Utility-based Dueling Banditscited by 31
  7. 2018PAC Battling Bandits in the Plackett-Luce Modelcited by 27
  8. 2024Feel-Good Thompson Sampling for Contextual Dueling Banditscited by 23
  9. 2023Variance-Aware Regret Bounds for Stochastic Contextual Dueling Banditscited by 22
  10. 2018Active Ranking with Subset-wise Preferencescited by 21
  11. 2022Federated Neural Banditscited by 17
  12. 2022Exploiting Correlation to Achieve Faster Learning Rates in Low-Rank Preference Banditscited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 5; 2026: 1.

7 papers.

  1. 2024Sharp Analysis for KL-Regularized Contextual Bandits and RLHFcited by 32
  2. 2025Large Language Model-Enhanced Multi-Armed Banditscited by 7
  3. 2025Neural Logistic Banditscited by 5
  4. 2025Quantum Lipschitz Banditscited by 4
  5. 2025Federated Linear Dueling Banditscited by 2
  6. 2025Online Clustering of Dueling Banditscited by 2
  7. 2026Linear and Neural Dueling Bandits with Delayed Feedbackcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Active Human Feedback Collection via Neural Contextual Dueling Bandits
  2. 2023Kernelized Offline Contextual Dueling Bandits
  3. 2024Nearly Optimal Algorithms for Contextual Dueling Bandits from Adversarial Feedback
  4. 2023Provable Benefits of Policy Learning from Human Preferences in Contextual Bandit Problems
  5. 2025Preference is More Than Comparisons: Rethinking Dueling Bandits with Augmented Human Feedback
  6. 2026Saturation-Insensitive Dueling Bandits with General Function Approximation
  7. 2023Contextual Bandits and Imitation Learning via Preference-Based Active Queries
  8. 2026On the Complexity of Preference-Based Bandits