mlsanityData to 7 Oct 2026

Trust, Don't Trust, or Flip: Robust Preference-Based Reinforcement Learning with Multi-Expert Feedback

arXiv 2601.18751 ↗First posted 26 Jan 2026cs.LG cs.AIcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Preference-based reinforcement learning (PBRL) offers a promising alternative to explicit reward engineering by learning from pairwise trajectory comparisons. However, real-world preference data often comes from heterogeneous annotators with varying reliability; some accurate, some noisy, and some systematically adversarial. Existing PBRL methods either treat all feedback equally or attempt to filter out unreliable sources, but both approaches fail when faced with adversarial annotators who systematically provide incorrect preferences. We introduce TriTrust-PBRL (TTP), a unified framework that jointly learns a shared reward model and expert-specific trust parameters from multi-expert preference feedback. The key insight is that trust parameters naturally evolve during gradient-based optimization to be positive (trust), near zero (ignore), or negative (flip), enabling the model to automatically invert adversarial preferences and recover useful signal rather than merely discarding corrupted feedback. We provide theoretical analysis establishing identifiability guarantees and detailed gradient analysis that explains how expert separation emerges naturally during training without explicit supervision. Empirically, we evaluate TTP on four diverse domains spanning manipulation tasks (MetaWorld) and locomotion (DM Control) under various corruption scenarios. TTP achieves state-of-the-art robustness, maintaining near-oracle performance under adversarial corruption while standard PBRL methods fail catastrophically. Notably, TTP outperforms existing baselines by successfully learning from mixed expert pools containing both reliable and adversarial annotators, all while requiring no expert features beyond identification indices and integrating seamlessly with existing PBRL pipelines.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 26 of its 37 references that are in the map. First posted on arXiv 26 Jan 2026.

Builds on

26 of 37 references.

Closest ancestors · 14

Same direction, newest first

  1. 2025Similarity as Reward Alignment: Robust and Versatile Preference-based Reinforcement Learning14 Jun
  2. 2025TREND: Tri-teaching for Robust Preference-based Reinforcement Learning with Demonstrations9 May
  3. 2025Strategyproof Reinforcement Learning from Human Feedback12 Mar
  4. 2025Robust Reward Alignment via Hypothesis Space Batch Cutting5 Feb
  5. 2024Rethinking Bradley-Terry Models in Preference-Based Reward Modeling: Foundations, Theory, and Alternatives7 Nov
  6. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning19 Aug
  7. 2024Robust Reinforcement Learning from Corrupted Human Feedback21 Jun
  8. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  9. 2022Reward Uncertainty for Exploration in Preference-based Reinforcement Learning24 May
  10. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  11. 2022SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learning18 Mar
  12. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training9 Jun
2 more
  1. 2018Reward learning from human preferences and demonstrations in Atari15 Nov
  2. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 12

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2021B-Pref: Benchmarking Preference-Based Reinforcement Learningcited by 106
  5. 2018Batch Active Preference-Based Learning of Reward Functionscited by 87
  6. 2019Asking Easy Questions: A User-Friendly Approach to Active Reward Learningcited by 67
  7. 2024Diverging Preferences: When do Annotators Disagree and do Models Know?cited by 47
  8. 2024RIME: Robust Preference-based Reinforcement Learning with Noisy Preferencescited by 37
  9. 2023RLHFPoison: Reward Poisoning Attack for Reinforcement Learning with Human Feedback in Large Language Modelscited by 32
  10. 2025From 1,000,000 Users to Every User: Scaling Up Personalized Preference for User-level Alignmentcited by 28
  11. 2024Beyond the Binary: Capturing Diverse Preferences With Reward Regularizationcited by 11
  12. 2025When Human Preferences Flip: An Instance-Dependent Robust Loss for RLHFcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Trust Region Reward Optimization and Proximal Inverse Reward Optimization Algorithm
  2. 2026From Reward-Free Representations to Preferences: Rethinking Offline Preference-Based Reinforcement Learning
  3. 2026Trust Region Inverse Reinforcement Learning: Explicit Dual Ascent using Local Policy Updates
  4. 2023Direct Preference-based Policy Optimization without Reward Modeling
  5. 2023Inverse Preference Learning: Preference-based RL without a Reward Function
  6. 2025Best Policy Learning from Trajectory Preference Feedback
  7. 2025Offline Safe Policy Optimization From Heterogeneous Feedback
  8. 2021Dueling RL: Reinforcement Learning with Trajectory Preferences