Learning Online from Corrective Feedback: A Meta-Algorithm for Robotics
On the map: Offline RL and imitation learning from data → Imitation learning methods
Abstract (arXiv)
A key challenge in Imitation Learning (IL) is that optimal state actions demonstrations are difficult for the teacher to provide. For example in robotics, providing kinesthetic demonstrations on a robotic manipulator requires the teacher to control multiple degrees of freedom at once. The difficulty of requiring optimal state action demonstrations limits the space of problems where the teacher can provide quality feedback. As an alternative to state action demonstrations, the teacher can provide corrective feedback such as their preferences or rewards. Prior work has created algorithms designed to learn from specific types of noisy feedback, but across teachers and tasks different forms of feedback may be required. Instead we propose that in order to learn from a diversity of scenarios we need to learn from a variety of feedback. To learn from a variety of feedback we make the following insight: the teacher's cost function is latent and we can model a stream of feedback as a stream of loss functions. We then use any online learning algorithm to minimize the sum of these losses. With this insight we can learn from a diversity of feedback that is weakly correlated with the teacher's true cost function. We unify prior work into a general corrective feedback meta-algorithm and show that regardless of feedback we can obtain the same regret bounds. We demonstrate our approach by learning to perform a household navigation task on a robotic racecar platform. Our results show that our approach can learn quickly from a variety of noisy feedback.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 9 of its 24 references that are in the map. First posted on arXiv 2 Apr 2021.
Builds on
9 of 24 references.
Closest ancestors · 1
Same direction, newest first
Other references · 8
- 2017Deep reinforcement learning from human preferencescited by 3,912
- 2018Gibson Env: Real-World Perception for Embodied Agentscited by 481
- 2017Interactive Learning from Policy-Dependent Human Feedbackcited by 158
- 2017Deeply AggreVaTeD: Differentiable Imitation Learning for Sequential Predictioncited by 93
- 2019Deep Reinforcement Learning from Policy-Dependent Human Feedbackcited by 56
- 2016Learning Preferences for Manipulation Tasks from Online Coactive Feedbackcited by 31
- 2019MuSHR: A Low-Cost, Open-Source Robotic Racecar for Education and Researchcited by 18
- 2018Including Uncertainty when Learning from Human Correctionscited by 15
Built on by
Citing papers per year: 2023: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2021Learning from Imperfect Demonstrations from Agents with Varying Dynamics
- 2025When a Robot is More Capable than a Human: Learning from Constrained Demonstrators
- 2021Ergodic imitation: Learning from what to do and what not to do
- 2022Imitation Learning by Estimating Expertise of Demonstrators
- 2025Action Chunking and Exploratory Data Collection Yield Exponential Improvements in Behavior Cloning for Continuous Control
- 2022Unified Learning from Demonstrations, Corrections, and Preferences during Physical Human-Robot Interaction
- 2019Watch, Try, Learn: Meta-Learning from Demonstrations and Reward
- 2020Scalable Multi-Task Imitation Learning with Autonomous Improvement