# Learning Preferences for Manipulation Tasks from Online Coactive Feedback

arXiv: https://arxiv.org/abs/1601.00741 · HTML: https://mlsanity.com/paper/1601.00741/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Jan 2016.

## Built on by (most cited first, 30 of 31)

- [An Algorithmic Perspective on Imitation Learning](https://mlsanity.com/paper/1811.06711/) (2018)
- [RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs](https://mlsanity.com/paper/2404.08555/) (2024)
- [Human-in-the-loop: Provably Efficient Preference-based Reinforcement Learning with General Function Approximation](https://mlsanity.com/paper/2205.11140/) (2022)
- [Correcting Robot Plans with Natural Language Feedback](https://mlsanity.com/paper/2204.05186/) (2022)
- [LATTE: LAnguage Trajectory TransformEr](https://mlsanity.com/paper/2208.02918/) (2022)
- [Humans are not Boltzmann Distributions: Challenges and Opportunities for Modelling Human Feedback and Interaction in Reinforcement Learning](https://mlsanity.com/paper/2206.13316/) (2022)
- [Efficient Preference-Based Reinforcement Learning Using Learned Dynamics Models](https://mlsanity.com/paper/2301.04741/) (2023)
- [Physical Interaction as Communication: Learning Robot Objectives Online from Human Corrections](https://mlsanity.com/paper/2107.02349/) (2021)
- [Unified Learning from Demonstrations, Corrections, and Preferences during Physical Human-Robot Interaction](https://mlsanity.com/paper/2207.03395/) (2022)
- [Inducing Structure in Reward Learning by Learning Features](https://mlsanity.com/paper/2201.07082/) (2022)
- [The Reasonable Crowd: Towards evidence-based and interpretable models of driving behavior](https://mlsanity.com/paper/2107.13507/) (2021)
- [Learning Perceptual Concepts by Bootstrapping from Human Queries](https://mlsanity.com/paper/2111.05251/) (2021)
- [Aligning Robot and Human Representations](https://mlsanity.com/paper/2302.01928/) (2023)
- [Batch Active Learning of Reward Functions from Human Preferences](https://mlsanity.com/paper/2402.15757/) (2024)
- [Pragmatic Feature Preferences: Learning Reward-Relevant Preferences from Human Input](https://mlsanity.com/paper/2405.14769/) (2024)
- [Eliciting User Preferences for Personalized Multi-Objective Decision Making through Comparative Feedback](https://mlsanity.com/paper/2302.03805/) (2023)
- [Assisted Robust Reward Design](https://mlsanity.com/paper/2111.09884/) (2021)
- [Goal Inference from Open-Ended Dialog](https://mlsanity.com/paper/2410.13957/) (2024)
- [Inverse reinforcement learning for autonomous navigation via differentiable semantic mapping and planning](https://mlsanity.com/paper/2101.00186/) (2021)
- [From Action Labels to Sets: Rethinking Action Supervision for Imitation Learning from Corrective Feedback](https://mlsanity.com/paper/2502.07645/) (2025)
- [Learning from Physical Human Feedback: An Object-Centric One-Shot Adaptation Method](https://mlsanity.com/paper/2203.04951/) (2022)
- [Evolutionary Retrofitting](https://mlsanity.com/paper/2410.11330/) (2024)
- [ProVox: Personalization and Proactive Planning for Situated Human-Robot Collaboration](https://mlsanity.com/paper/2506.12248/) (2025)
- [Preference Aligned Visuomotor Diffusion Policies for Deformable Object Manipulation](https://mlsanity.com/paper/2602.09583/) (2026)
- [Learning Online from Corrective Feedback: A Meta-Algorithm for Robotics](https://mlsanity.com/paper/2104.01021/) (2021)
- [Learning from Preferences and Mixed Demonstrations in General Settings](https://mlsanity.com/paper/2508.14027/) (2025)
- [Flexible Agent Alignment with Goal Inference from Open-Ended Dialog](https://mlsanity.com/paper/2508.15119/) (2025)
- [Enhancing Goal Inference via Correction Timing](https://mlsanity.com/paper/2602.18603/) (2026)
- [Offline Constrained RLHF with Multiple Preference Oracles](https://mlsanity.com/paper/2604.00200/) (2026)
- [Aligning Robot Representations with Humans](https://mlsanity.com/paper/2205.07882/) (2022)

## Abstract (arXiv)

We consider the problem of learning preferences over trajectories for mobile manipulators such as personal robots and assembly line robots. The preferences we learn are more intricate than simple geometric constraints on trajectories; they are rather governed by the surrounding context of various objects and human interactions in the environment. We propose a coactive online learning framework for teaching preferences in contextually rich environments. The key novelty of our approach lies in the type of feedback expected from the user: the human user does not need to demonstrate optimal trajectories as training data, but merely needs to iteratively provide trajectories that slightly improve over the trajectory currently proposed by the system. We argue that this coactive preference feedback can be more easily elicited than demonstrations of optimal trajectories. Nevertheless, theoretical regret bounds of our algorithm match the asymptotic rates of optimal trajectory algorithms. We implement our algorithm on two high degree-of-freedom robots, PR2 and Baxter, and present three intuitive mechanisms for providing such incremental feedback. In our experimental evaluation we consider two context rich settings -- household chores and grocery store checkout -- and show that users are able to train the robot with just a few feedbacks (taking only a few minutes).footnoteParts of this work has been published at NIPS and ISRR conferences~citepJain13,Jain13b. This journal submission presents a consistent full paper, and also includes the proof of regret bounds, more details of the robotic system, and a thorough related work.