# Learning Human Objectives from Sequences of Physical Corrections

arXiv: https://arxiv.org/abs/2104.00078 · HTML: https://mlsanity.com/paper/2104.00078/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Mar 2021.

## Built on by (most cited first, 21 of 21)

- [Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2307.15217/) (2023)
- [RT-H: Action Hierarchies Using Language](https://mlsanity.com/paper/2403.01823/) (2024)
- [Few-Shot Preference Learning for Human-in-the-Loop RL](https://mlsanity.com/paper/2212.03363/) (2022)
- ["No, to the Right" -- Online Language Corrections for Robotic Manipulation via Shared Autonomy](https://mlsanity.com/paper/2301.02555/) (2023)
- [Learning Multimodal Rewards from Rankings](https://mlsanity.com/paper/2109.12750/) (2021)
- [Learning Reward Functions from Scale Feedback](https://mlsanity.com/paper/2110.00284/) (2021)
- [Physical Interaction as Communication: Learning Robot Objectives Online from Human Corrections](https://mlsanity.com/paper/2107.02349/) (2021)
- [Unified Learning from Demonstrations, Corrections, and Preferences during Physical Human-Robot Interaction](https://mlsanity.com/paper/2207.03395/) (2022)
- [APReL: A Library for Active Preference-based Reward Learning Algorithms](https://mlsanity.com/paper/2108.07259/) (2021)
- [Active Reward Learning from Online Preferences](https://mlsanity.com/paper/2302.13507/) (2023)
- [Toward Grounded Commonsense Reasoning](https://mlsanity.com/paper/2306.08651/) (2023)
- [HERD: Continuous Human-to-Robot Evolution for Learning from Human Demonstration](https://mlsanity.com/paper/2212.04359/) (2022)
- [Contrastive Learning from Exploratory Actions: Leveraging Natural Interactions for Preference Elicitation](https://mlsanity.com/paper/2501.01367/) (2025)
- [Anticipatory Planning: Improving Long-Lived Planning by Estimating Expected Cost of Future Tasks](https://mlsanity.com/paper/2305.04692/) (2023)
- [Leveraging Human Revisions for Improving Text-to-Layout Models](https://mlsanity.com/paper/2405.13026/) (2024)
- [Coaching a Robotic Sonographer: Learning Robotic Ultrasound with Sparse Expert's Feedback](https://mlsanity.com/paper/2409.02337/) (2024)
- [Bayesian inference for data-efficient, explainable, and safe robotic motion planning: A review](https://mlsanity.com/paper/2307.08024/) (2023)
- [Efficiently Generating Expressive Quadruped Behaviors via Language-Guided Preference Learning](https://mlsanity.com/paper/2502.03717/) (2025)
- [Combining Bayesian Inference and Reinforcement Learning for Agent Decision Making: A Review](https://mlsanity.com/paper/2505.07911/) (2025)
- [Residual Reward Models: Leveraging Prior Knowledge for Efficient Preference-based Reinforcement Learning in Robotics](https://mlsanity.com/paper/2507.00611/) (2025)
- [The Power of Combined Modalities in Interactive Robot Learning](https://mlsanity.com/paper/2405.07817/) (2024)

## Abstract (arXiv)

When personal, assistive, and interactive robots make mistakes, humans naturally and intuitively correct those mistakes through physical interaction. In simple situations, one correction is sufficient to convey what the human wants. But when humans are working with multiple robots or the robot is performing an intricate task often the human must make several corrections to fix the robot's behavior. Prior research assumes each of these physical corrections are independent events, and learns from them one-at-a-time. However, this misses out on crucial information: each of these interactions are interconnected, and may only make sense if viewed together. Alternatively, other work reasons over the final trajectory produced by all of the human's corrections. But this method must wait until the end of the task to learn from corrections, as opposed to inferring from the corrections in an online fashion. In this paper we formalize an approach for learning from sequences of physical corrections during the current task. To do this we introduce an auxiliary reward that captures the human's trade-off between making corrections which improve the robot's immediate reward and long-term performance. We evaluate the resulting algorithm in remote and in-person human-robot experiments, and compare to both independent and final baselines. Our results indicate that users are best able to convey their objective when the robot reasons over their sequence of corrections.