mlsanityData to 14 Oct 2026

Active Human Feedback Collection via Neural Contextual Dueling Bandits

arXiv 2504.12016 ↗First posted 16 Apr 2025cs.LGcited by 4

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Collecting human preference feedback is often expensive, leading recent works to develop principled algorithms to select them more efficiently. However, these works assume that the underlying reward function is linear, an assumption that does not hold in many real-life applications, such as online recommendation and LLM alignment. To address this limitation, we propose Neural-ADB, an algorithm based on the neural contextual dueling bandit framework that provides a principled and practical method for collecting human preference feedback when the underlying latent reward function is non-linear. We theoretically show that when preference feedback follows the Bradley-Terry-Luce model, the worst sub-optimality gap of the policy learned by Neural-ADB decreases at a sub-linear rate as the preference dataset increases. Our experimental results on preference datasets further corroborate the effectiveness of Neural-ADB.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 31 of its 61 references that are in the map. First posted on arXiv 16 Apr 2025.

Builds on

31 of 61 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs12 Apr
  2. 2024Reinforcement Learning from Human Feedback with Active Queries14 Feb
  3. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons26 Jan
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr

Other references · 27

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  2. 2017On Kernelized Multi-armed Banditscited by 352
  3. 2022Teaching language models to support answers with verified quotescited by 292
  4. 2018Deep Bayesian Bandits Showdown: An Empirical Comparison of Bayesian Deep Networks for Thompson Samplingcited by 169
  5. 2020Neural Thompson Samplingcited by 128
  6. 2018Preference-based Online Learning with Dueling Bandits: A Surveycited by 102
  7. 2020Neural Contextual Bandits with Deep Representation and Shallow Explorationcited by 71
  8. 2014Reducing Dueling Bandits to Cardinal Banditscited by 62
  9. 2015Regret Lower Bound and Optimal Algorithm in Dueling Bandit Problemcited by 52
  10. 2021EE-Net: Exploitation-Exploration Neural Networks in Contextual Banditscited by 44
  11. 2021Efficient and Optimal Algorithms for Contextual Dueling Bandits under Realizabilitycited by 44
  12. 2022Stochastic Contextual Dueling Bandits under Linear Stochastic Transitivity Modelscited by 37
15 more
  1. 2024Prompt Optimization with Human Feedbackcited by 35
  2. 2016A Relative Exponential Weighing Algorithm for Adversarial Utility-based Dueling Banditscited by 31
  3. 2021Personalized News Recommendation: Methods and Challengescited by 30
  4. 2018PAC Battling Bandits in the Plackett-Luce Modelcited by 27
  5. 2024Feel-Good Thompson Sampling for Contextual Dueling Banditscited by 27
  6. 2023Prompt Learning for News Recommendationcited by 25
  7. 2023Variance-Aware Regret Bounds for Stochastic Contextual Dueling Banditscited by 22
  8. 2018Active Ranking with Subset-wise Preferencescited by 21
  9. 2022Federated Neural Banditscited by 17
  10. 2019Online Algorithm for Unsupervised Sensor Selectioncited by 7
  11. 2020Online Algorithm for Unsupervised Sequential Selection with Contextual Informationcited by 7
  12. 2023Contextual Bandits with Online Neural Regressioncited by 7
  13. 2022Bandits for Online Calibration: An Application to Content Moderation on Social Media Platformscited by 6
  14. 2024Conversational Dueling Bandits in Generalized Linear Modelscited by 5
  15. 2020Thompson Sampling for Unsupervised Sequential Selectioncited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

  1. 2025ActiveDPO: Active Direct Preference Optimization for Sample-Efficient Alignmentcited by 10
  2. 2026Robust AI Evaluation through Maximal Lotteriescited by 1
  3. 2025Uncovering Scaling Laws for Large Language Models via Inverse Problemscited by 0
  4. 2026Stochastic Multi-Armed Bandits with Limited Control Variatescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Neural Dueling Bandits: Preference-Based Optimization with Human Feedback
  2. 2025Preference is More Than Comparisons: Rethinking Dueling Bandits with Augmented Human Feedback
  3. 2026Linear and Neural Dueling Bandits with Delayed Feedback
  4. 2023Provable Benefits of Policy Learning from Human Preferences in Contextual Bandit Problems
  5. 2023Kernelized Offline Contextual Dueling Bandits
  6. 2023Sample Efficient Preference Alignment in LLMs via Active Exploration
  7. 2025Contextual bandits with entropy-based human feedback
  8. 2024Enhancing Preference-based Linear Bandits via Human Response Time