mlsanityData to 7 Oct 2026

Linear and Neural Dueling Bandits with Delayed Feedback

arXiv 2605.26554 ↗First posted 26 May 2026cs.LG cs.AIcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Contextual dueling bandits form a cornerstone of preference-based decision-making, with critical applications in recommender systems and large language model alignment. However, standard algorithms rely on the idealized assumption of immediate feedback, a condition frequently violated in real-world scenarios such as prompt optimization. This setting introduces a unique theoretical challenge: unlike linear bandits, dueling bandit estimators lack closed-form solutions, rendering naive adaptations of standard weighting techniques biased. To address this, we formalize the problem of Contextual Dueling Bandits with Stochastic Delayed Feedback and propose two novel algorithms: Linear (LDB-DF) and Neural (NDB-DF) Dueling Bandits with Delayed Feedback. Central to our approach is a novel estimator that integrates an Inverse Probability Weighting (IPW) mechanism directly into the loss function, ensuring unbiased correction for delayed or missing feedback. We provide comprehensive theoretical analysis, establishing an O(d*sqrt(T)) regret bound for the linear setting and sub-linear guarantees for the neural setting. Extensive experiments on both simulated and real-world datasets demonstrate the effectiveness of our propose.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 18 of its 28 references that are in the map. First posted on arXiv 26 May 2026.

Builds on

18 of 28 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Neural Dueling Bandits: Preference-Based Optimization with Human Feedback24 Jul
  2. 2024Nearly Optimal Algorithms for Contextual Dueling Bandits from Adversarial Feedback16 Apr

Other references · 16

  1. 2023Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolutioncited by 521
  2. 2013Online Learning under Delayed Feedbackcited by 137
  3. 2020Neural Thompson Samplingcited by 127
  4. 2023InstructZero: Efficient Instruction Optimization for Black-Box Large Language Modelscited by 68
  5. 2017Stochastic Bandit Models for Delayed Conversionscited by 53
  6. 2015Regret Lower Bound and Optimal Algorithm in Dueling Bandit Problemcited by 51
  7. 2021Efficient and Optimal Algorithms for Contextual Dueling Bandits under Realizabilitycited by 44
  8. 2022Stochastic Contextual Dueling Bandits under Linear Stochastic Transitivity Modelscited by 35
  9. 2024Prompt Optimization with Human Feedbackcited by 35
  10. 2018Best arm identification in multi-armed bandits with delayed feedbackcited by 29
  11. 2018Learning Contextual Bandits in a Non-stationary Environmentcited by 26
  12. 2022Sample-Then-Optimize Batch Neural Thompson Samplingcited by 25
4 more
  1. 2024Feel-Good Thompson Sampling for Contextual Dueling Banditscited by 23
  2. 2022Bayesian Optimization under Stochastic Delayed Feedbackcited by 10
  3. 2019Stochastic Bandits with Delayed Composite Anonymous Feedbackcited by 6
  4. 2025Online Clustering of Dueling Banditscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Kernelized Offline Contextual Dueling Bandits
  2. 2025Active Human Feedback Collection via Neural Contextual Dueling Bandits
  3. 2026Saturation-Insensitive Dueling Bandits with General Function Approximation
  4. 2025Preference is More Than Comparisons: Rethinking Dueling Bandits with Augmented Human Feedback
  5. 2024Learning from Imperfect Human Feedback: a Tale from Corruption-Robust Dueling
  6. 2026Multi-User Dueling Bandits: A Fair Approach using Nash Social Welfare
  7. 2024Beyond Numeric Rewards: In-Context Dueling Bandits with LLM Agents
  8. 2024Bandits with Preference Feedback: A Stackelberg Game Perspective