mlsanityData to 7 Oct 2026

Boosting Robustness in Preference-Based Reinforcement Learning with Dynamic Sparsity

arXiv 2406.06495 ↗First posted 10 Jun 2024cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

To integrate into human-centered environments, autonomous agents must learn from and adapt to humans in their native settings. Preference-based reinforcement learning (PbRL) can enable this by learning reward functions from human preferences. However, humans live in a world full of diverse information, most of which is irrelevant to completing any particular task. It then becomes essential that agents learn to focus on the subset of task-relevant state features. To that end, this work proposes R2N (Robust-to-Noise), the first PbRL algorithm that leverages principles of dynamic sparse training to learn robust reward models that can focus on task-relevant features. In experiments with a simulated teacher, we demonstrate that R2N can adapt the sparse connectivity of its neural networks to focus on task-relevant features, enabling R2N to significantly outperform several sparse training and PbRL algorithms across simulated robotic environments.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 28 of its 45 references that are in the map. First posted on arXiv 10 Jun 2024.

Builds on

28 of 45 references.

References · 28

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  5. 2017Deep reinforcement learning from human preferencescited by 3,846
  6. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  7. 2018DeepMind Control Suitecited by 935
  8. 2019Rigging the Lottery: Making All Tickets Winnerscited by 491
  9. 2017Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Sciencecited by 325
  10. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 298
  11. 2019Sparse Networks from Scratch: Faster Training without Losing Performancecited by 210
  12. 2017Interactive Learning from Policy-Dependent Human Feedbackcited by 155
16 more
  1. 2017Deep Rewiring: Training very sparse deep networkscited by 136
  2. 2017Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spacescited by 122
  3. 2021B-Pref: Benchmarking Preference-Based Reinforcement Learningcited by 106
  4. 2022SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learningcited by 96
  5. 2022Few-Shot Preference Learning for Human-in-the-Loop RLcited by 85
  6. 2021MEST: Accurate and Fast Memory-Economic Sparse Training Framework on the Edgecited by 77
  7. 2023Inverse Preference Learning: Preference-based RL without a Reward Functioncited by 68
  8. 2022The State of Sparse Training in Deep Reinforcement Learningcited by 60
  9. 2019Discovering Neural Wiringscited by 56
  10. 2021Dynamic Sparse Training for Deep Reinforcement Learningcited by 49
  11. 2019A Hitchhiker's Guide to Statistical Comparisons of Reinforcement Learning Algorithmscited by 26
  12. 2023Rating-based Reinforcement Learningcited by 14
  13. 2023Automatic Noise Filtering with Dynamic Sparse Training in Deep Reinforcement Learningcited by 13
  14. 2019Improving Deep Reinforcement Learning in Minecraft with Action Advicecited by 10
  15. 2019A Brain-inspired Algorithm for Training Highly Sparse Neural Networkscited by 8
  16. 2024Leveraging Sub-Optimal Data for Human-in-the-Loop Reinforcement Learningcited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Robust Reinforcement Learning from Corrupted Human Feedback
  2. 2024RIME: Robust Preference-based Reinforcement Learning with Noisy Preferences
  3. 2025Preference-Guided Learning for Sparse-Reward Multi-Agent Reinforcement Learning
  4. 2023Contrastive Preference Learning: Learning from Human Feedback without RL
  5. 2023Direct Preference-based Policy Optimization without Reward Modeling
  6. 2025Residual Reward Models: Leveraging Prior Knowledge for Efficient Preference-based Reinforcement Learning in Robotics
  7. 2024Adaptive Preference Scaling for Reinforcement Learning with Human Feedback
  8. 2025On Symmetric Losses for Robust Policy Optimization with Noisy Preferences