mlsanityData to 7 Oct 2026

Robust Reinforcement Learning from Corrupted Human Feedback

arXiv 2406.15568 ↗First posted 21 Jun 2024cs.LGcited by 25

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reinforcement learning from human feedback (RLHF) provides a principled framework for aligning AI systems with human preference data. For various reasons, e.g., personal bias, context ambiguity, lack of training, etc, human annotators may give incorrect or inconsistent preference labels. To tackle this challenge, we propose a robust RLHF approach -- R3M, which models the potentially corrupted preference label as sparse outliers. Accordingly, we formulate the robust reward learning as an ℓ1-regularized maximum likelihood estimation problem. Computationally, we develop an efficient alternating optimization algorithm, which only incurs negligible computational overhead compared with the standard RLHF approach. Theoretically, we prove that under proper regularity conditions, R3M can consistently learn the underlying reward and identify outliers, provided that the number of outlier labels scales sublinearly with the preference sample size. Furthermore, we remark that R3M is versatile and can be extended to various preference optimization methods, including direct preference optimization (DPO). Our experiments on robotic control and natural language generation with large language models (LLMs) show that R3M improves robustness of the reward against several types of perturbations to the preference data.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 29 of its 47 references that are in the map. First posted on arXiv 21 Jun 2024.

Builds on

29 of 47 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024Corruption Robust Offline Reinforcement Learning with Human Feedback9 Feb
  2. 2023Reward Model Ensembles Help Mitigate Overoptimization4 Oct
  3. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback27 Jul
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  5. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedback17 May
  6. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons26 Jan
  7. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  9. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 20

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  5. 2020Learning to summarize from human feedbackcited by 2,562
  6. 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,510
  7. 2018Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labelscited by 1,053
  8. 2016Adversarial Machine Learning at Scalecited by 934
  9. 2017Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learningcited by 918
  10. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  11. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  12. 2017A General and Adaptive Robust Loss Functioncited by 236
8 more
  1. 2021A Universal Law of Robustness via Isoperimetrycited by 150
  2. 2015Distributionally Robust Logistic Regressioncited by 124
  3. 2021B-Pref: Benchmarking Preference-Based Reinforcement Learningcited by 106
  4. 2021Sample Selection with Uncertainty of Losses for Learning with Noisy Labelscited by 79
  5. 2020Smooth Exploration for Robotic Reinforcement Learningcited by 45
  6. 2024RIME: Robust Preference-based Reinforcement Learning with Noisy Preferencescited by 37
  7. 2023Robust Multi-Agent Reinforcement Learning via Adversarial Regularization: Theoretical Foundation and Stable Algorithmscited by 25
  8. 2020Doubly Robust Off-Policy Learning on Low-Dimensional Manifolds by Deep Neural Networkscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 15; 2026: 10.

25 papers.

Sort:

By area:

  1. 2025Robust LLM Alignment via Distributionally Robust Direct Preference Optimizationcited by 17
  2. 2025Think-RM: Enabling Long-Horizon Reasoning in Generative Reward Modelscited by 15
  3. 2025Robust Reinforcement Learning from Human Feedback for Large Language Models Fine-Tuningcited by 13
  4. 2025Distributionally Robust Reinforcement Learning with Human Feedbackcited by 12
  5. 2025Noise-corrected GRPO: From Noisy Rewards to Unbiased Gradientscited by 11
  6. 2025A Unified Theoretical Analysis of Private and Robust Offline Alignment: from RLHF to DPOcited by 8
  7. 2025Clone-Robust AI Alignmentcited by 7
  8. 2025Strategyproof Reinforcement Learning from Human Feedbackcited by 5
  9. 2025Two Minds Better Than One: Collaborative Reward Modeling for LLM Alignmentcited by 5
  10. 2025Scalable Valuation of Human Feedback through Provably Robust Model Alignmentcited by 4
  11. 2025Interaction-Driven Browsing: A Human-in-the-Loop Conceptual Framework Informed by Human Web Browsing for Browser-Using Agentscited by 3
  12. 2025When Human Preferences Flip: An Instance-Dependent Robust Loss for RLHFcited by 3
13 more
  1. 2025Lightweight Robust Direct Preference Optimizationcited by 2
  2. 2025Provably Mitigating Corruption, Overoptimization, and Verbosity Simultaneously in Offline and Online RLHF/DPO Alignmentcited by 1
  3. 2026DeDPO: Debiased Direct Preference Optimization for Diffusion Modelscited by 1
  4. 2026A Jointly Efficient and Optimal Algorithm for Heteroskedastic Generalized Linear Bandits with Adversarial Corruptionscited by 1
  5. 2026Regret Bounds for Reinforcement Learning from Multi-Source Imperfect Preferencescited by 1
  6. 2025Towards Integrated Alignmentcited by 0
  7. 2026Trust, Don't Trust, or Flip: Robust Preference-Based Reinforcement Learning with Multi-Expert Feedbackcited by 0
  8. 2026Aligning to Illusions: Choice Blindness in Human and AI Feedbackcited by 0
  9. 2026Inverse Reinforcement Learning without an Optimal Demonstrator: A Feasible Reward Set Approachcited by 0
  10. 2026Provably Efficient Personalized Multi-Objective Bandits with Proactive Conversational Queriescited by 0
  11. 2026Metadata-Free Meta-Reweighted Direct Preference Optimization under Noisy Preference Labelscited by 0
  12. 2026Improving Generalization and Robustness in Offline Reinforcement Learning via Boundary-Aware Data Augmentationcited by 0
  13. 2026Robust Risk-Sensitive Reinforcement Learning from Corrupted Human Feedbackcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  2. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint
  3. 2023Contrastive Preference Learning: Learning from Human Feedback without RL
  4. 2024Adaptive Preference Scaling for Reinforcement Learning with Human Feedback
  5. 2025Doubly Robust Alignment for Large Language Models
  6. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning
  7. 2025On Symmetric Losses for Robust Policy Optimization with Noisy Preferences
  8. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedback