mlsanityData to 14 Oct 2026

Corruption Robust Offline Reinforcement Learning with Human Feedback

arXiv 2402.06734 ↗First posted 9 Feb 2024cs.LG cs.AIcited by 20

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

also in 1 more
Abstract (arXiv)

We study data corruption robustness for reinforcement learning with human feedback (RLHF) in an offline setting. Given an offline dataset of pairs of trajectories along with feedback about human preferences, an varepsilon-fraction of the pairs is corrupted (e.g., feedback flipped or trajectory features manipulated), capturing an adversarial attack or noisy human preferences. We aim to design algorithms that identify a near-optimal policy from the corrupted data, with provable guarantees. Existing theoretical works have separately studied the settings of corruption robust RL (learning from scalar rewards directly under corruption) and offline RLHF (learning from human feedback without corruption); however, they are inapplicable to our problem of dealing with corrupted data in offline RLHF setting. To this end, we design novel corruption robust offline RLHF methods under various assumptions on the coverage of the data-generating distributions. At a high level, our methodology robustifies an offline RLHF framework by first learning a reward model along with confidence sets and then learning a pessimistic optimal policy over the confidence set. Our key insight is that learning optimal policy can be done by leveraging an offline corruption-robust RL oracle in different ways (e.g., zero-order oracle or first-order oracle), depending on the data coverage assumptions. To our knowledge, ours is the first work that provides provable corruption robust offline RLHF methods.

Key facts. Cited by 20 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 38 of its 49 references that are in the map. First posted on arXiv 9 Feb 2024.

Builds on

38 of 49 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024Crowd-PrefRL: Preference-Based Reward Learning from Crowds17 Jan
  2. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback27 Jul
  3. 2023Is RLHF More Difficult than Standard RL?25 Jun
  4. 2023Reinforcement Learning from Diverse Human Preferences27 Jan
  5. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons26 Jan
  6. 2023Benchmarks and Algorithms for Offline Preference-Based Reward Learning3 Jan
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  8. 2019Fine-Tuning Language Models from Human Preferences18 Sep
  9. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 29

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2020Learning to summarize from human feedbackcited by 2,588
  3. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  4. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 590
  5. 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 520
  6. 2020Is Pessimism Provably Efficient for Offline RL?cited by 364
  7. 2013Optimal rates for zero-order convex optimization: the power of two function evaluationscited by 198
  8. 2019A Short Note on Concentration Inequalities for Random Vectors with SubGaussian Normcited by 106
  9. 2021B-Pref: Benchmarking Preference-Based Reinforcement Learningcited by 106
  10. 2021Dueling RL: Reinforcement Learning with Trajectory Preferencescited by 106
  11. 2019Recent Advances in Algorithmic High-Dimensional Robust Statisticscited by 101
  12. 2022When Is Partially Observable Reinforcement Learning Not Scary?cited by 100
17 more
  1. 2022Robust Reinforcement Learning using Offline Datacited by 96
  2. 2017Being Robust (in High Dimensions) Can Be Practicalcited by 83
  3. 2020Policy Teaching via Environment Poisoning: Training-time Adversarial Attacks against Reinforcement Learningcited by 81
  4. 2022Human-in-the-loop: Provably Efficient Preference-based Reinforcement Learning with General Function Approximationcited by 81
  5. 2019Corruption-robust exploration in episodic reinforcement learningcited by 72
  6. 2021Corruption-Robust Offline Reinforcement Learningcited by 51
  7. 2020Outlier Robust Mean Estimation with Subgaussian Rates via Stabilitycited by 50
  8. 2021A Model Selection Approach for Corruption Robust Reinforcement Learningcited by 45
  9. 2019Generalized Resilience and Robust Statisticscited by 37
  10. 2022Corruption-Robust Algorithms with Uncertainty Weighting for Nonlinear Contextual Bandits and Markov Decision Processescited by 33
  11. 2021On the Theory of Reinforcement Learning with Once-per-Episode Feedbackcited by 32
  12. 2023Corruption-Robust Offline Reinforcement Learning with General Function Approximationcited by 31
  13. 2021Robust Policy Gradient against Strong Data Corruptioncited by 30
  14. 2022COPA: Certifying Robust Policies for Offline Reinforcement Learning against Poisoning Attackscited by 26
  15. 2023Towards Robust Offline Reinforcement Learning under Diverse Data Corruptioncited by 24
  16. 2021Improved Corruption Robust Algorithms for Episodic Reinforcement Learningcited by 20
  17. 2023Offline Primal-Dual Reinforcement Learning for Linear MDPscited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 11; 2026: 6.

20 papers.

Sort:

By area:

  1. 2024Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMscited by 142
  2. 2024Robust Reinforcement Learning from Corrupted Human Feedbackcited by 25
  3. 2025Robust Reinforcement Learning from Human Feedback for Large Language Models Fine-Tuningcited by 13
  4. 2025Distributionally Robust Reinforcement Learning with Human Feedbackcited by 12
  5. 2025Clone-Robust AI Alignmentcited by 9
  6. 2025A Unified Theoretical Analysis of Private and Robust Offline Alignment: from RLHF to DPOcited by 8
  7. 2024Tackling Data Corruption in Offline Reinforcement Learning via Sequence Modelingcited by 7
  8. 2025Strategyproof Reinforcement Learning from Human Feedbackcited by 5
  9. 2025Scalable Valuation of Human Feedback through Provably Robust Model Alignmentcited by 4
  10. 2025SquarechiPO: Differentially Private and Robust chi2-Preference Optimization in Offline Direct Alignmentcited by 3
  11. 2025Policy Teaching via Data Poisoning in Learning from Human Preferencescited by 2
  12. 2025Corruption-Tolerant Asynchronous Q-Learning with Near-Optimal Ratescited by 1
8 more
  1. 2025Provably Mitigating Corruption, Overoptimization, and Verbosity Simultaneously in Offline and Online RLHF/DPO Alignmentcited by 1
  2. 2026A Jointly Efficient and Optimal Algorithm for Heteroskedastic Generalized Linear Bandits with Adversarial Corruptionscited by 1
  3. 2026Regret Bounds for Reinforcement Learning from Multi-Source Imperfect Preferencescited by 1
  4. 2026Efficient Preference Poisoning Attack on Offline RLHFcited by 1
  5. 2025Sparse Offline Reinforcement Learning with Corruption Robustnesscited by 0
  6. 2026Corruption-robust Offline Multi-agent Reinforcement Learning From Human Feedbackcited by 0
  7. 2026RMB: Reward Model Boosting Mitigates Reward Hackingcited by 0
  8. 2026Robust Risk-Sensitive Reinforcement Learning from Corrupted Human Feedbackcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Enhancing Robustness of Offline Reinforcement Learning Under Data Corruption via Sharpness-Aware Minimization
  2. 2024Towards Robust Model-Based Reinforcement Learning Against Adversarial Corruption
  3. 2023Robust Offline Reinforcement Learning -- Certify the Confidence Interval
  4. 2025Robust Policy Expansion for Offline-to-Online RL under Diverse Data Corruption
  5. 2024Learning from Imperfect Human Feedback: a Tale from Corruption-Robust Dueling
  6. 2024Towards Robust Policy: Enhancing Offline Reinforcement Learning with Adversarial Attacks and Defenses
  7. 2024Online Bandit Learning with Offline Preference Data for Improved RLHF
  8. 2022RORL: Robust Offline Reinforcement Learning via Conservative Smoothing