# RIME: Robust Preference-based Reinforcement Learning with Noisy Preferences

arXiv: https://arxiv.org/abs/2402.17257 · HTML: https://mlsanity.com/paper/2402.17257/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 30 of its 47 references that are in the map.
- First posted on arXiv 27 Feb 2024.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 30 of 30)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning](https://mlsanity.com/paper/1910.10897/) (2019)
- [Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels](https://mlsanity.com/paper/1805.07836/) (2018)
- [DeepMind Control Suite](https://mlsanity.com/paper/1801.00690/) (2018)
- [Learning from Noisy Labels with Deep Neural Networks: A Survey](https://mlsanity.com/paper/2007.08199/) (2020)
- [Robust Loss Functions under Label Noise for Deep Neural Networks](https://mlsanity.com/paper/1712.09482/) (2017)
- [Aligning Text-to-Image Models using Human Feedback](https://mlsanity.com/paper/2302.12192/) (2023)
- [dm_control: Software and Tasks for Continuous Control](https://mlsanity.com/paper/2006.12983/) (2020)
- [Reward learning from human preferences and demonstrations in Atari](https://mlsanity.com/paper/1811.06521/) (2018)
- [PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training](https://mlsanity.com/paper/2106.05091/) (2021)
- [Does label smoothing mitigate label noise?](https://mlsanity.com/paper/2003.02819/) (2020)
- [Behavior From the Void: Unsupervised Active Pre-Training](https://mlsanity.com/paper/2103.04551/) (2021)
- [Gradient Descent with Early Stopping is Provably Robust to Label Noise for Overparameterized Neural Networks](https://mlsanity.com/paper/1903.11680/) (2019)
- [Mastering the Game of Stratego with Model-Free Multiagent Reinforcement Learning](https://mlsanity.com/paper/2206.15378/) (2022)
- [SMARTS: Scalable Multi-Agent Reinforcement Learning Training School for Autonomous Driving](https://mlsanity.com/paper/2010.09776/) (2020)
- [Towards Human-Level Bimanual Dexterous Manipulation with Reinforcement Learning](https://mlsanity.com/paper/2206.08686/) (2022)
- [Jump-Start Reinforcement Learning](https://mlsanity.com/paper/2204.02372/) (2022)
- [B-Pref: Benchmarking Preference-Based Reinforcement Learning](https://mlsanity.com/paper/2111.03026/) (2021)
- [SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learning](https://mlsanity.com/paper/2203.10050/) (2022)
- [Curriculum Loss: Robust Learning and Generalization against Label Corruption](https://mlsanity.com/paper/1905.10045/) (2019)
- [Few-Shot Preference Learning for Human-in-the-Loop RL](https://mlsanity.com/paper/2212.03363/) (2022)
- [Denoising Implicit Feedback for Recommendation](https://mlsanity.com/paper/2006.04153/) (2020)
- [Preference Transformer: Modeling Human Preferences using Transformers for RL](https://mlsanity.com/paper/2303.00957/) (2023)
- [To Smooth or Not? When Label Smoothing Meets Noisy Labels](https://mlsanity.com/paper/2106.04149/) (2021)
- [Reincarnating Reinforcement Learning: Reusing Prior Computation to Accelerate Progress](https://mlsanity.com/paper/2206.01626/) (2022)
- [Reward Uncertainty for Exploration in Preference-based Reinforcement Learning](https://mlsanity.com/paper/2205.12401/) (2022)
- [Weakly Supervised Learning with Side Information for Noisy Labeled Images](https://mlsanity.com/paper/2008.11586/) (2020)
- [Reinforcement Learning from Diverse Human Preferences](https://mlsanity.com/paper/2301.11774/) (2023)
- [Robust Learning by Self-Transition for Handling Noisy Labels](https://mlsanity.com/paper/2012.04337/) (2020)

## Built on by (most cited first, 30 of 39)

- [Robust Reinforcement Learning from Corrupted Human Feedback](https://mlsanity.com/paper/2406.15568/) (2024)
- [Ada-K Routing: Boosting the Efficiency of MoE-based LLMs](https://mlsanity.com/paper/2410.10456/) (2024)
- [Scaling Offline Model-Based RL via Jointly-Optimized World-Action Model Pretraining](https://mlsanity.com/paper/2410.00564/) (2024)
- [Skill Expansion and Composition in Parameter Space](https://mlsanity.com/paper/2502.05932/) (2025)
- [VARP: Reinforcement Learning from Vision-Language Model Feedback with Agent Regularized Preferences](https://mlsanity.com/paper/2503.13817/) (2025)
- [VRPO: Rethinking Value Modeling for Robust RL under Noisy Supervision in LLM Post-Training](https://mlsanity.com/paper/2508.03058/) (2025)
- [Preference VLM: Leveraging VLMs for Scalable Preference-Based Reinforcement Learning](https://mlsanity.com/paper/2502.01616/) (2025)
- [TREND: Tri-teaching for Robust Preference-based Reinforcement Learning with Demonstrations](https://mlsanity.com/paper/2505.06079/) (2025)
- [Leveraging Sub-Optimal Data for Human-in-the-Loop Reinforcement Learning](https://mlsanity.com/paper/2405.00746/) (2024)
- [Toward Stable World Models: Measuring and Addressing World Instability in Generative Environments](https://mlsanity.com/paper/2503.08122/) (2025)
- [Strategyproof Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2503.09561/) (2025)
- [S-EPOA: Overcoming the Indistinguishability of Segments with Skill-Driven Preference-Based Reinforcement Learning](https://mlsanity.com/paper/2408.12130/) (2024)
- [Policy Learning from Large Vision-Language Model Feedback without Reward Modeling](https://mlsanity.com/paper/2507.23391/) (2025)
- [CLARIFY: Contrastive Preference Reinforcement Learning for Untangling Ambiguous Queries](https://mlsanity.com/paper/2506.00388/) (2025)
- [Aligning Humans and Robots via Reinforcement Learning from Implicit Human Feedback](https://mlsanity.com/paper/2507.13171/) (2025)
- [STAIR: Addressing Stage Misalignment through Temporal-Aligned Preference Reinforcement Learning](https://mlsanity.com/paper/2509.23802/) (2025)
- [When Human Preferences Flip: An Instance-Dependent Robust Loss for RLHF](https://mlsanity.com/paper/2512.00709/) (2025)
- [Enhancing Reinforcement Learning for Radiology Report Generation with Evidence-aware Rewards and Self-correcting Preference Learning](https://mlsanity.com/paper/2604.13598/) (2026)
- [Robust Reward Alignment via Hypothesis Space Batch Cutting](https://mlsanity.com/paper/2502.02921/) (2025)
- [Similarity as Reward Alignment: Robust and Versatile Preference-based Reinforcement Learning](https://mlsanity.com/paper/2506.12529/) (2025)
- [Evaluating Feature Dependent Noise in Preference-based Reinforcement Learning](https://mlsanity.com/paper/2601.01904/) (2026)
- [RoDiF: Robust Direct Fine-Tuning of Diffusion Policies with Corrupted Human Feedback](https://mlsanity.com/paper/2602.00886/) (2026)
- [DAPPER: Discriminability-Aware Policy-to-Policy Preference-Based Reinforcement Learning for Query-Efficient Robot Skill Acquisition](https://mlsanity.com/paper/2505.06357/) (2025)
- [Provably Mitigating Corruption, Overoptimization, and Verbosity Simultaneously in Offline and Online RLHF/DPO Alignment](https://mlsanity.com/paper/2510.05526/) (2025)
- [Explainable reinforcement learning from human feedback to improve alignment](https://mlsanity.com/paper/2512.13837/) (2025)
- [Reducing Oracle Feedback with Vision-Language Embeddings for Preference-Based RL](https://mlsanity.com/paper/2603.28053/) (2026)
- [Pseudo-Quantized Actor-Critic Algorithm for Robustness to Noisy Temporal Difference Error](https://mlsanity.com/paper/2604.01613/) (2026)
- [PrefMoE: Robust Preference Modeling with Mixture-of-Experts Reward Learning](https://mlsanity.com/paper/2605.00384/) (2026)
- [Efficient Preference Poisoning Attack on Offline RLHF](https://mlsanity.com/paper/2605.02495/) (2026)
- [COLLIE: Guiding Skill Discovery in Semantically Coherent Latent Space](https://mlsanity.com/paper/2606.00950/) (2026)

## Abstract (arXiv)

Preference-based Reinforcement Learning (PbRL) circumvents the need for reward engineering by harnessing human preferences as the reward signal. However, current PbRL methods excessively depend on high-quality feedback from domain experts, which results in a lack of robustness. In this paper, we present RIME, a robust PbRL algorithm for effective reward learning from noisy preferences. Our method utilizes a sample selection-based discriminator to dynamically filter out noise and ensure robust training. To counteract the cumulative error stemming from incorrect selection, we suggest a warm start for the reward model, which additionally bridges the performance gap during the transition from pre-training to online training in PbRL. Our experiments on robotic manipulation and locomotion tasks demonstrate that RIME significantly enhances the robustness of the state-of-the-art PbRL method. Code is available at https://github.com/CJReinforce/RIME_ICML2024.