mlsanityData to 14 Oct 2026

Learning from Imperfect Human Feedback: a Tale from Corruption-Robust Dueling

arXiv 2405.11204 ↗First posted 18 May 2024cs.LG stat.MLcited by 2

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

This paper studies Learning from Imperfect Human Feedback (LIHF), addressing the potential irrationality or imperfect perception when learning from comparative human feedback. Building on evidences that human's imperfection decays over time (i.e., humans learn to improve), we cast this problem as a concave-utility continuous-action dueling bandit but under a restricted form of corruption: i.e., the corruption scale is decaying over time as tρ-1 for some "imperfection rate" ρ ∈ [0, 1]. With T as the total number of iterations, we establish a regret lower bound of Omega(max√T, Tρ) for LIHF, even when ρ is known. For the same setting, we develop the Robustified Stochastic Mirror Descent for Imperfect Dueling (RoSMID) algorithm, which achieves nearly optimal regret tildeO(max√T, Tρ). Core to our analysis is a novel framework for analyzing gradient-based algorithms for dueling bandit under corruption, and we demonstrate its general applicability by showing how this framework can be easily applied to obtain corruption-robust guarantees for other popular gradient-based dueling bandit algorithms. Our theoretical results are validated by extensive experiments.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 25 of its 53 references that are in the map. First posted on arXiv 18 May 2024.

Builds on

25 of 53 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Nearly Optimal Algorithms for Contextual Dueling Bandits from Adversarial Feedback16 Apr
  2. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback27 Jul
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr

Other references · 22

  1. 2020Learning to summarize from human feedbackcited by 2,588
  2. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  3. 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,517
  4. 2018Robustness May Be at Odds with Accuracycited by 880
  5. 2018Stochastic bandits robust to adversarial corruptionscited by 119
  6. 2018Preference-based Online Learning with Dueling Bandits: A Surveycited by 102
  7. 2019Better Algorithms for Stochastic Bandits with Adversarial Corruptionscited by 92
  8. 2020Stochastic Linear Bandits Robust to Adversarial Attackscited by 65
  9. 2014Reducing Dueling Bandits to Cardinal Banditscited by 62
  10. 2015Regret Lower Bound and Optimal Algorithm in Dueling Bandit Problemcited by 52
  11. 2022Nearly Optimal Algorithms for Linear Contextual Bandits with Adversarial Corruptionscited by 51
  12. 2021Robust Stochastic Linear Contextual Bandits Under Adversarial Attackscited by 33
10 more
  1. 2022Corruption-Robust Algorithms with Uncertainty Weighting for Nonlinear Contextual Bandits and Markov Decision Processescited by 33
  2. 2018Large-scale Interactive Recommendation with Tree-structured Policy Gradientcited by 30
  3. 2018Incentivizing Exploration with Selective Data Disclosurecited by 24
  4. 2013Linear Bandits in High Dimension and Recommendation Systemscited by 22
  5. 2017Regret Analysis for Continuous Dueling Banditcited by 22
  6. 2023Variance-Aware Regret Bounds for Stochastic Contextual Dueling Banditscited by 22
  7. 2024Principled Preferential Bayesian Optimizationcited by 16
  8. 2022A Robust Phased Elimination Algorithm for Corruption-Tolerant Gaussian Process Banditscited by 11
  9. 2021Learning the Optimal Recommendation from Explorative Userscited by 3
  10. 2022Learning from a Learning User for Optimal Recommendationscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025Robust Reinforcement Learning from Human Feedback for Large Language Models Fine-Tuningcited by 13
  2. 2026Robust Risk-Sensitive Reinforcement Learning from Corrupted Human Feedbackcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Corruption Robust Offline Reinforcement Learning with Human Feedback
  2. 2024Neural Dueling Bandits: Preference-Based Optimization with Human Feedback
  3. 2023Kernelized Offline Contextual Dueling Bandits
  4. 2024Corruption-Robust Linear Bandits: Minimax Optimality and Gap-Dependent Misspecification
  5. 2026Multi-User Dueling Bandits: A Fair Approach using Nash Social Welfare
  6. 2026Regret Bounds for Reinforcement Learning from Multi-Source Imperfect Preferences
  7. 2024Robust Reinforcement Learning from Corrupted Human Feedback
  8. 2026Linear and Neural Dueling Bandits with Delayed Feedback