mlsanityData to 14 Oct 2026

CANDERE-COACH: Reinforcement Learning from Noisy Feedback

arXiv 2409.15521 ↗First posted 23 Sep 2024cs.LG cs.AIcited by 3

On the map: Offline RL and imitation learning from data → Imitation learning methods

also in 1 more
Abstract (arXiv)

In recent times, Reinforcement learning (RL) has been widely applied to many challenging tasks. However, in order to perform well, it requires access to a good reward function which is often sparse or manually engineered with scope for error. Introducing human prior knowledge is often seen as a possible solution to the above-mentioned problem, such as imitation learning, learning from preference, and inverse reinforcement learning. Learning from feedback is another framework that enables an RL agent to learn from binary evaluative signals describing the teacher's (positive or negative) evaluation of the agent's action. However, these methods often make the assumption that evaluative teacher feedback is perfect, which is a restrictive assumption. In practice, such feedback can be noisy due to limited teacher expertise or other exacerbating factors like cognitive load, availability, distraction, etc. In this work, we propose the CANDERE-COACH algorithm, which is capable of learning from noisy feedback by a nonoptimal teacher. We propose a noise-filtering mechanism to de-noise online feedback data, thereby enabling the RL agent to successfully learn with up to 40% of the teacher feedback being incorrect. Experiments on three common domains demonstrate the effectiveness of the proposed approach.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 18 of its 36 references that are in the map. First posted on arXiv 23 Sep 2024.

Builds on

18 of 36 references.

References · 18

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2017Deep reinforcement learning from human preferencescited by 3,912
  3. 2016Generative Adversarial Imitation Learningcited by 1,597
  4. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,204
  5. 2018Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labelscited by 1,059
  6. 2019Solving Rubik's Cube with a Robot Handcited by 734
  7. 2017MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labelscited by 653
  8. 2020Learning from Noisy Labels with Deep Neural Networks: A Surveycited by 582
  9. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 302
  10. 2021How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learnedcited by 213
  11. 2017Interactive Learning from Policy-Dependent Human Feedbackcited by 158
  12. 2017Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spacescited by 123
6 more
  1. 2021Reward (Mis)design for Autonomous Drivingcited by 104
  2. 2020Learning from Suboptimal Demonstration via Self-Supervised Reward Regressioncited by 84
  3. 2019Deep Reinforcement Learning from Policy-Dependent Human Feedbackcited by 56
  4. 2020Model-Based Inverse Reinforcement Learning from Visual Demonstrationscited by 52
  5. 2024Correlated Proxies: A New Definition and Improved Mitigation for Reward Hackingcited by 50
  6. 2021Action Advising with Advice Imitation in Deep Reinforcement Learningcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025Robust Reward Alignment via Hypothesis Space Batch Cuttingcited by 2
  2. 2026RoDiF: Robust Direct Fine-Tuning of Diffusion Policies with Corrupted Human Feedbackcited by 2
  3. 2026Reinforcement Learning for Computer-Use Agents with Autonomous Evaluationcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026Behavior Cloning is Not All You Need: The Optimality of On-Policy Distillation for Noisy Expert Feedback
  2. 2024Navigating Noisy Feedback: Enhancing Reinforcement Learning with Error-Prone Language Models
  3. 2021Learning Online from Corrective Feedback: A Meta-Algorithm for Robotics
  4. 2022The Expertise Problem: Learning from Specialized Feedback
  5. 2025Reinforcement Learning via Implicit Imitation Guidance
  6. 2023Good Better Best: Self-Motivated Imitation Learning for noisy Demonstrations
  7. 2025Reinforcement Learning from Human Feedback
  8. 2021A Closer Look at Advantage-Filtered Behavioral Cloning in High-Noise Datasets