# Learning Preferences for Interactive Autonomy

arXiv: https://arxiv.org/abs/2210.10899 · HTML: https://mlsanity.com/paper/2210.10899/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 226 references that are in the map.
- First posted on arXiv 19 Oct 2022.

## Builds on (most cited first, 21 of 21)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Safe Controller Optimization for Quadrotors with Gaussian Processes](https://mlsanity.com/paper/1509.01066/) (2015)
- [Hierarchical Game-Theoretic Planning for Autonomous Vehicles](https://mlsanity.com/paper/1810.05766/) (2018)
- [Batch Active Learning Using Determinantal Point Processes](https://mlsanity.com/paper/1906.07975/) (2019)
- [Learning from Imperfect Demonstrations from Agents with Varying Dynamics](https://mlsanity.com/paper/2103.05910/) (2021)
- [Human irrationality: both bad and good for reward inference](https://mlsanity.com/paper/2111.06956/) (2021)
- [Dissimilarity-based Sparse Subset Selection](https://mlsanity.com/paper/1407.6810/) (2014)
- [Learning from Richer Human Guidance: Augmenting Comparison-Based Learning with Feature Queries](https://mlsanity.com/paper/1802.01604/) (2018)
- [The Green Choice: Learning and Influencing Human Decisions on Shared Roads](https://mlsanity.com/paper/1904.02209/) (2019)
- [Characterizing Driving Styles with Deep Learning](https://mlsanity.com/paper/1607.03611/) (2016)
- [Learning an Urban Air Mobility Encounter Model from Expert Preferences](https://mlsanity.com/paper/1907.05575/) (2019)
- [Learning Constraints from Demonstrations](https://mlsanity.com/paper/1812.07084/) (2018)
- [Interactive Robot Training for Non-Markov Tasks](https://mlsanity.com/paper/2003.02232/) (2020)
- [Preference-based Learning of Reward Function Features](https://mlsanity.com/paper/2103.02727/) (2021)
- [Leveraging Smooth Attention Prior for Multi-Agent Trajectory Prediction](https://mlsanity.com/paper/2203.04421/) (2022)
- [Altruistic Autonomy: Beating Congestion on Shared Roads](https://mlsanity.com/paper/1810.11978/) (2018)
- [Incentivizing Efficient Equilibria in Traffic Networks with Mixed Autonomy](https://mlsanity.com/paper/2106.04678/) (2021)
- [Do You Want Your Autonomous Car To Drive Like You?](https://mlsanity.com/paper/1802.01636/) (2018)
- [How do people incorporate advice from artificial agents when making physical judgments?](https://mlsanity.com/paper/2205.11613/) (2022)

## Built on by (most cited first, 6 of 6)

- [Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2307.15217/) (2023)
- [Robots That Ask For Help: Uncertainty Alignment for Large Language Model Planners](https://mlsanity.com/paper/2307.01928/) (2023)
- [Rank2Reward: Learning Shaped Reward Functions from Passive Video](https://mlsanity.com/paper/2404.14735/) (2024)
- [Breadcrumbs to the Goal: Goal-Conditioned Exploration from Human-in-the-Loop Feedback](https://mlsanity.com/paper/2307.11049/) (2023)
- [Autonomous Robotic Reinforcement Learning with Asynchronous Human Feedback](https://mlsanity.com/paper/2310.20608/) (2023)
- [Interactive Groupwise Comparison for Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2507.04340/) (2025)

## Abstract (arXiv)

When robots enter everyday human environments, they need to understand their tasks and how they should perform those tasks. To encode these, reward functions, which specify the objective of a robot, are employed. However, designing reward functions can be extremely challenging for complex tasks and environments. A promising approach is to learn reward functions from humans. Recently, several robot learning works embrace this approach and leverage human demonstrations to learn the reward functions. Known as inverse reinforcement learning, this approach relies on a fundamental assumption: humans can provide near-optimal demonstrations to the robot. Unfortunately, this is rarely the case: human demonstrations to the robot are often suboptimal due to various reasons, e.g., difficulty of teleoperation, robot having high degrees of freedom, or humans' cognitive limitations. This thesis is an attempt towards learning reward functions from human users by using other, more reliable data modalities. Specifically, we study how reward functions can be learned using comparative feedback, in which the human user compares multiple robot trajectories instead of (or in addition to) providing demonstrations. To this end, we first propose various forms of comparative feedback, e.g., pairwise comparisons, best-of-many choices, rankings, scaled comparisons; and describe how a robot can use these various forms of human feedback to infer a reward function, which may be parametric or non-parametric. Next, we propose active learning techniques to enable the robot to ask for comparison feedback that optimizes for the expected information that will be gained from that user feedback. Finally, we demonstrate the applicability of our methods in a wide variety of domains, ranging from autonomous driving simulations to home robotics, from standard reinforcement learning benchmarks to lower-body exoskeletons.