mlsanityData to 7 Oct 2026

Corruption-robust Offline Multi-agent Reinforcement Learning From Human Feedback

arXiv 2603.28281 ↗First posted 30 Mar 2026cs.LGcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

also in 1 more
Abstract (arXiv)

We consider robustness against data corruption in offline multi-agent reinforcement learning from human feedback (MARLHF) under a strong-contamination model: given a dataset D of trajectory-preference tuples (each preference being an n-dimensional binary label vector representing each of the n agents' preferences), an ε-fraction of the samples may be arbitrarily corrupted. We model the problem using the framework of linear Markov games. First, under a uniform coverage assumption - where every policy of interest is sufficiently represented in the clean (prior to corruption) data - we introduce a robust estimator that guarantees an O(ε1 - o(1)) bound on the Nash equilibrium gap. Next, we move to the more challenging unilateral coverage setting, in which only a Nash equilibrium and its single-player deviations are covered. In this case, our proposed algorithm achieves an O(√ε) bound on the Nash gap. Both of these procedures, however, suffer from intractable computation. To address this, we relax our solution concept to coarse correlated equilibria (CCE). Under the same unilateral coverage regime, we derive a quasi-polynomial-time algorithm whose CCE gap scales as O(√ε). To the best of our knowledge, this is the first systematic treatment of adversarial data corruption in offline MARLHF.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 45 of its 61 references that are in the map. First posted on arXiv 30 Mar 2026.

Builds on

45 of 61 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferences4 Mar
  2. 2024Corruption Robust Offline Reinforcement Learning with Human Feedback9 Feb
  3. 2023Reinforcement Learning with Human Feedback: Learning Dynamic Choices via Pessimism29 May
  4. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons26 Jan
  5. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  7. 2021WebGPT: Browser-assisted question-answering with human feedback17 Dec
  8. 2019Fine-Tuning Language Models from Human Preferences18 Sep
  9. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 36

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2020Learning to summarize from human feedbackcited by 2,562
  3. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  4. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  5. 2017Adversarial Attacks on Neural Network Policiescited by 314
  6. 2022Teaching language models to support answers with verified quotescited by 290
  7. 2021Recursively Summarizing Books with Human Feedbackcited by 287
  8. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 256
  9. 2016Robust Estimators in High Dimensions without the Computational Intractabilitycited by 209
  10. 2019Adversarial Policies: Attacking Deep Reinforcement Learningcited by 209
  11. 2017Tactics of Adversarial Attack on Deep Reinforcement Learning Agentscited by 163
  12. 2017Interactive Learning from Policy-Dependent Human Feedbackcited by 155
24 more
  1. 2017Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spacescited by 122
  2. 2023Universal Jailbreak Backdoors from Poisoned Human Feedbackcited by 120
  3. 2021V-Learning -- A Simple, Efficient, Decentralized Algorithm for Multiagent RLcited by 111
  4. 2018Is feature selection secure against training data poisoning?cited by 88
  5. 2021Near-Optimal No-Regret Learning in General Gamescited by 85
  6. 2023BadGPT: Exploring Security Vulnerabilities of ChatGPT via Backdoor Attacks to InstructGPTcited by 70
  7. 2021When Can We Learn General-Sum Markov Games with a Large Number of Players Sample-Efficiently?cited by 63
  8. 2021Corruption-Robust Offline Reinforcement Learningcited by 51
  9. 2020Outlier Robust Mean Estimation with Subgaussian Rates via Stabilitycited by 50
  10. 2022Pessimistic Minimax Value Iteration: Provably Efficient Equilibrium Learning from Offline Datasetscited by 42
  11. 2019Quantum Entropy Scoring for Fast Robust Mean Estimation and Improved Outlier Detectioncited by 41
  12. 2024Best-of-Venom: Attacking RLHF by Injecting Poisoned Preference Datacited by 33
  13. 2023Corruption-Robust Offline Reinforcement Learning with General Function Approximationcited by 30
  14. 2022Provably Efficient Offline Multi-agent Reinforcement Learning via Strategy-wise Bonuscited by 29
  15. 2023Breaking the Curse of Multiagency: Provably Efficient Decentralized Multi-Agent RL with Function Approximationcited by 29
  16. 2023Breaking the Curse of Multiagents in a Large State Space: RL in Markov Games with Independent Linear Function Approximationcited by 26
  17. 2020Policy Teaching in Reinforcement Learning via Environment Poisoning Attackscited by 24
  18. 2022Understanding the Limits of Poisoning Attacks in Episodic Reinforcement Learningcited by 21
  19. 2022Reward Poisoning Attacks on Offline Multi-Agent Reinforcement Learningcited by 19
  20. 2023Offline Learning in Markov Games with General Function Approximationcited by 16
  21. 2024SoS Certifiability of Subgaussian Distributions and its Algorithmic Applicationscited by 10
  22. 2023Hardness of Independent Learning and Sparse Equilibrium Computation in Markov Gamescited by 9
  23. 2024Corruption-Robust Offline Two-Player Zero-Sum Markov Gamescited by 4
  24. 2025Policy Teaching via Data Poisoning in Learning from Human Preferencescited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Sparse Offline Reinforcement Learning with Corruption Robustness
  2. 2023Towards Robust Offline Reinforcement Learning under Diverse Data Corruption
  3. 2025Enhancing Robustness of Offline Reinforcement Learning Under Data Corruption via Sharpness-Aware Minimization
  4. 2024Tackling Data Corruption in Offline Reinforcement Learning via Sequence Modeling
  5. 2024Robust Reinforcement Learning from Corrupted Human Feedback
  6. 2024Robust Offline Reinforcement Learning for Non-Markovian Decision Processes
  7. 2022Distributionally Robust Model-Based Offline Reinforcement Learning with Near-Optimal Sample Complexity
  8. 2024Uncertainty-based Offline Variational Bayesian Reinforcement Learning for Robustness under Diverse Data Corruptions