mlsanityData to 14 Oct 2026

RIME: Robust Preference-based Reinforcement Learning with Noisy Preferences

arXiv 2402.17257 ↗First posted 27 Feb 2024cs.LG cs.AI cs.ROcited by 39

On the map: Deep reinforcement learning methods → RL training of language and agent models

Abstract (arXiv)

Preference-based Reinforcement Learning (PbRL) circumvents the need for reward engineering by harnessing human preferences as the reward signal. However, current PbRL methods excessively depend on high-quality feedback from domain experts, which results in a lack of robustness. In this paper, we present RIME, a robust PbRL algorithm for effective reward learning from noisy preferences. Our method utilizes a sample selection-based discriminator to dynamically filter out noise and ensure robust training. To counteract the cumulative error stemming from incorrect selection, we suggest a warm start for the reward model, which additionally bridges the performance gap during the transition from pre-training to online training in PbRL. Our experiments on robotic manipulation and locomotion tasks demonstrate that RIME significantly enhances the robustness of the state-of-the-art PbRL method. Code is available at https://github.com/CJReinforce/RIME_ICML2024.

Key facts. Cited by 39 papers in the map (data through 14 Oct 2026). Placed in RL training of language and agent models, one of 15 directions in Deep reinforcement learning methods. Builds on 30 of its 47 references that are in the map. First posted on arXiv 27 Feb 2024.

Builds on

30 of 47 references.

References · 30

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Deep reinforcement learning from human preferencescited by 3,912
  3. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,204
  4. 2018Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labelscited by 997
  5. 2018DeepMind Control Suitecited by 956
  6. 2020Learning from Noisy Labels with Deep Neural Networks: A Surveycited by 582
  7. 2017Robust Loss Functions under Label Noise for Deep Neural Networkscited by 443
  8. 2023Aligning Text-to-Image Models using Human Feedbackcited by 422
  9. 2020dm_control: Software and Tasks for Continuous Controlcited by 375
  10. 2018Reward learning from human preferences and demonstrations in Ataricited by 316
  11. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 302
  12. 2020Does label smoothing mitigate label noise?cited by 204
18 more
  1. 2021Behavior From the Void: Unsupervised Active Pre-Trainingcited by 174
  2. 2019Gradient Descent with Early Stopping is Provably Robust to Label Noise for Overparameterized Neural Networkscited by 159
  3. 2022Mastering the Game of Stratego with Model-Free Multiagent Reinforcement Learningcited by 131
  4. 2020SMARTS: Scalable Multi-Agent Reinforcement Learning Training School for Autonomous Drivingcited by 121
  5. 2022Towards Human-Level Bimanual Dexterous Manipulation with Reinforcement Learningcited by 119
  6. 2022Jump-Start Reinforcement Learningcited by 112
  7. 2021B-Pref: Benchmarking Preference-Based Reinforcement Learningcited by 106
  8. 2022SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learningcited by 97
  9. 2019Curriculum Loss: Robust Learning and Generalization against Label Corruptioncited by 88
  10. 2022Few-Shot Preference Learning for Human-in-the-Loop RLcited by 87
  11. 2020Denoising Implicit Feedback for Recommendationcited by 80
  12. 2023Preference Transformer: Modeling Human Preferences using Transformers for RLcited by 79
  13. 2021To Smooth or Not? When Label Smoothing Meets Noisy Labelscited by 70
  14. 2022Reincarnating Reinforcement Learning: Reusing Prior Computation to Accelerate Progresscited by 65
  15. 2022Reward Uncertainty for Exploration in Preference-based Reinforcement Learningcited by 61
  16. 2020Weakly Supervised Learning with Side Information for Noisy Labeled Imagescited by 30
  17. 2023Reinforcement Learning from Diverse Human Preferencescited by 24
  18. 2020Robust Learning by Self-Transition for Handling Noisy Labelscited by 18

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 20; 2026: 14.

39 papers.

Sort:

By area:

  1. 2024Robust Reinforcement Learning from Corrupted Human Feedbackcited by 25
  2. 2024Ada-K Routing: Boosting the Efficiency of MoE-based LLMscited by 16
  3. 2024Scaling Offline Model-Based RL via Jointly-Optimized World-Action Model Pretrainingcited by 14
  4. 2025Skill Expansion and Composition in Parameter Spacecited by 14
  5. 2025VARP: Reinforcement Learning from Vision-Language Model Feedback with Agent Regularized Preferencescited by 9
  6. 2025VRPO: Rethinking Value Modeling for Robust RL under Noisy Supervision in LLM Post-Trainingcited by 9
  7. 2025Preference VLM: Leveraging VLMs for Scalable Preference-Based Reinforcement Learningcited by 7
  8. 2025TREND: Tri-teaching for Robust Preference-based Reinforcement Learning with Demonstrationscited by 7
  9. 2024Leveraging Sub-Optimal Data for Human-in-the-Loop Reinforcement Learningcited by 6
  10. 2025Toward Stable World Models: Measuring and Addressing World Instability in Generative Environmentscited by 5
  11. 2025Strategyproof Reinforcement Learning from Human Feedbackcited by 5
  12. 2024S-EPOA: Overcoming the Indistinguishability of Segments with Skill-Driven Preference-Based Reinforcement Learningcited by 4
27 more
  1. 2025Policy Learning from Large Vision-Language Model Feedback without Reward Modelingcited by 4
  2. 2025CLARIFY: Contrastive Preference Reinforcement Learning for Untangling Ambiguous Queriescited by 3
  3. 2025Aligning Humans and Robots via Reinforcement Learning from Implicit Human Feedbackcited by 3
  4. 2025STAIR: Addressing Stage Misalignment through Temporal-Aligned Preference Reinforcement Learningcited by 3
  5. 2025When Human Preferences Flip: An Instance-Dependent Robust Loss for RLHFcited by 3
  6. 2026Enhancing Reinforcement Learning for Radiology Report Generation with Evidence-aware Rewards and Self-correcting Preference Learningcited by 3
  7. 2025Robust Reward Alignment via Hypothesis Space Batch Cuttingcited by 2
  8. 2025Similarity as Reward Alignment: Robust and Versatile Preference-based Reinforcement Learningcited by 2
  9. 2026Evaluating Feature Dependent Noise in Preference-based Reinforcement Learningcited by 2
  10. 2026RoDiF: Robust Direct Fine-Tuning of Diffusion Policies with Corrupted Human Feedbackcited by 2
  11. 2025DAPPER: Discriminability-Aware Policy-to-Policy Preference-Based Reinforcement Learning for Query-Efficient Robot Skill Acquisitioncited by 1
  12. 2025Provably Mitigating Corruption, Overoptimization, and Verbosity Simultaneously in Offline and Online RLHF/DPO Alignmentcited by 1
  13. 2025Explainable reinforcement learning from human feedback to improve alignmentcited by 1
  14. 2026Reducing Oracle Feedback with Vision-Language Embeddings for Preference-Based RLcited by 1
  15. 2026Pseudo-Quantized Actor-Critic Algorithm for Robustness to Noisy Temporal Difference Errorcited by 1
  16. 2026PrefMoE: Robust Preference Modeling with Mixture-of-Experts Reward Learningcited by 1
  17. 2026Efficient Preference Poisoning Attack on Offline RLHFcited by 1
  18. 2026COLLIE: Guiding Skill Discovery in Semantically Coherent Latent Spacecited by 1
  19. 2026From Reward-Free Representations to Preferences: Rethinking Offline Preference-Based Reinforcement Learningcited by 1
  20. 2025Robustness in the Face of Partial Identifiability in Reward Learningcited by 0
  21. 2025PB2: Preference Space Exploration via Population-Based Methods in Preference-Based Reinforcement Learningcited by 0
  22. 2025Reinforcement Learning from Implicit Neural Feedback for Human-Aligned Robot Controlcited by 0
  23. 2026The Trajectory Alignment Coefficient in Two Acts: From Reward Tuning to Reward Learningcited by 0
  24. 2026Trust, Don't Trust, or Flip: Robust Preference-Based Reinforcement Learning with Multi-Expert Feedbackcited by 0
  25. 2026CoRe: Combined Rewards with Vision-Language Model Feedback for Preference-Aligned Reinforcement Learningcited by 0
  26. 2026To Go Far, Go Together: Diverse Preferences Induce a Curriculum for Reward Optimizationcited by 0
  27. 2026ALIGN-HOLD: Experience Alignment for Real-Time Hold Control in Large-Scale Ride-Hailing Matching at DiDicited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Direct Preference-based Policy Optimization without Reward Modeling
  2. 2024Multi-Type Preference Learning: Empowering Preference-Based Reinforcement Learning with Equal Preferences
  3. 2025FLoRA: Sample-Efficient Preference-based RL via Low-Rank Style Adaptation of Reward Functions
  4. 2024Advances in Preference-based Reinforcement Learning: A Review
  5. 2024Sample-Efficient Preference-based Reinforcement Learning with Dynamics Aware Rewards
  6. 2024Boosting Robustness in Preference-Based Reinforcement Learning with Dynamic Sparsity
  7. 2023Provable Reward-Agnostic Preference-Based Reinforcement Learning
  8. 2025Residual Reward Models: Leveraging Prior Knowledge for Efficient Preference-based Reinforcement Learning in Robotics