# Active Preference Learning using Maximum Regret

arXiv: https://arxiv.org/abs/2005.04067 · HTML: https://mlsanity.com/paper/2005.04067/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 May 2020.

## Built on by (most cited first, 21 of 21)

- [Reinforcement Learning from Human Feedback with Active Queries](https://mlsanity.com/paper/2402.09401/) (2024)
- [Learning Multimodal Rewards from Rankings](https://mlsanity.com/paper/2109.12750/) (2021)
- [Information Directed Reward Learning for Reinforcement Learning](https://mlsanity.com/paper/2102.12466/) (2021)
- [Learning Reward Functions from Scale Feedback](https://mlsanity.com/paper/2110.00284/) (2021)
- [Provably Feedback-Efficient Reinforcement Learning via Active Reward Learning](https://mlsanity.com/paper/2304.08944/) (2023)
- [APReL: A Library for Active Preference-based Reward Learning Algorithms](https://mlsanity.com/paper/2108.07259/) (2021)
- [Active Reward Learning from Online Preferences](https://mlsanity.com/paper/2302.13507/) (2023)
- [Preference-based Learning of Reward Function Features](https://mlsanity.com/paper/2103.02727/) (2021)
- [Batch Active Learning of Reward Functions from Human Preferences](https://mlsanity.com/paper/2402.15757/) (2024)
- [MoDiPO: text-to-motion alignment via AI-feedback-driven Direct Preference Optimization](https://mlsanity.com/paper/2405.03803/) (2024)
- [Optimistic Task Inference for Behavior Foundation Models](https://mlsanity.com/paper/2510.20264/) (2025)
- [Online Decision Mediation](https://mlsanity.com/paper/2310.18601/) (2023)
- [Preference Elicitation for Offline Reinforcement Learning](https://mlsanity.com/paper/2406.18450/) (2024)
- [Causally Robust Reward Learning from Reason-Augmented Preference Feedback](https://mlsanity.com/paper/2603.04861/) (2026)
- [A Generalized Acquisition Function for Preference-based Reward Learning](https://mlsanity.com/paper/2403.06003/) (2024)
- [Reactive Multi-Robot Navigation in Outdoor Environments Through Uncertainty-Aware Active Learning of Human Preference Landscape](https://mlsanity.com/paper/2409.16577/) (2024)
- [Experience in Engineering Complex Systems: Active Preference Learning with Multiple Outcomes and Certainty Levels](https://mlsanity.com/paper/2302.14630/) (2023)
- [Online Policy Learning from Offline Preferences](https://mlsanity.com/paper/2403.10160/) (2024)
- [Representation Alignment from Human Feedback for Cross-Embodiment Reward Learning from Mixed-Quality Demonstrations](https://mlsanity.com/paper/2408.05610/) (2024)
- [Automata Learning of Preferences over Temporal Logic Formulas from Pairwise Comparisons](https://mlsanity.com/paper/2505.18030/) (2025)
- [Which Rewards Matter? Reward Selection for Reinforcement Learning under Limited Feedback](https://mlsanity.com/paper/2510.00144/) (2025)

## Abstract (arXiv)

We study active preference learning as a framework for intuitively specifying the behaviour of autonomous robots. In active preference learning, a user chooses the preferred behaviour from a set of alternatives, from which the robot learns the user's preferences, modeled as a parameterized cost function. Previous approaches present users with alternatives that minimize the uncertainty over the parameters of the cost function. However, different parameters might lead to the same optimal behaviour; as a consequence the solution space is more structured than the parameter space. We exploit this by proposing a query selection that greedily reduces the maximum error ratio over the solution space. In simulations we demonstrate that the proposed approach outperforms other state of the art techniques in both learning efficiency and ease of queries for the user. Finally, we show that evaluating the learning based on the similarities of solutions instead of the similarities of weights allows for better predictions for different scenarios.