# Learning from Richer Human Guidance: Augmenting Comparison-Based Learning with Feature Queries

arXiv: https://arxiv.org/abs/1802.01604 · HTML: https://mlsanity.com/paper/1802.01604/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Feb 2018.

## Built on by (most cited first, 16 of 16)

- [Learning Multimodal Rewards from Rankings](https://mlsanity.com/paper/2109.12750/) (2021)
- [Learning Reward Functions from Scale Feedback](https://mlsanity.com/paper/2110.00284/) (2021)
- [Leveraging Approximate Symbolic Models for Reinforcement Learning via Skill Diversity](https://mlsanity.com/paper/2202.02886/) (2022)
- [How to talk so AI will learn: Instructions, descriptions, and autonomy](https://mlsanity.com/paper/2206.07870/) (2022)
- [Professional Agents -- Evolving Large Language Models into Autonomous Experts with Human-Level Competencies](https://mlsanity.com/paper/2402.03628/) (2024)
- [Pragmatic Feature Preferences: Learning Reward-Relevant Preferences from Human Input](https://mlsanity.com/paper/2405.14769/) (2024)
- [Linguistic communication as (inverse) reward design](https://mlsanity.com/paper/2204.05091/) (2022)
- [Learning Preferences for Interactive Autonomy](https://mlsanity.com/paper/2210.10899/) (2022)
- [MAPLE: A Framework for Active Preference Learning Guided by Large Language Models](https://mlsanity.com/paper/2412.07207/) (2024)
- [CIVIL: Causal and Intuitive Visual Imitation Learning](https://mlsanity.com/paper/2504.17959/) (2025)
- [Causally Robust Reward Learning from Reason-Augmented Preference Feedback](https://mlsanity.com/paper/2603.04861/) (2026)
- [The Linguistic Blind Spot of Value-Aligned Agency, Natural and Artificial](https://mlsanity.com/paper/2207.00868/) (2022)
- [Tell me why: Training preferences-based RL with human preferences and step-level explanations](https://mlsanity.com/paper/2405.14244/) (2024)
- [Freeform Preference Learning for Robotic Manipulation](https://mlsanity.com/paper/2606.32027/) (2026)
- [Online Behavior Modification for Expressive User Control of RL-Trained Robots](https://mlsanity.com/paper/2408.16776/) (2024)
- [Hierarchical Reward Design from Language: Enhancing Alignment of Agent Behavior with Human Specifications](https://mlsanity.com/paper/2602.18582/) (2026)

## Abstract (arXiv)

We focus on learning the desired objective function for a robot. Although trajectory demonstrations can be very informative of the desired objective, they can also be difficult for users to provide. Answers to comparison queries, asking which of two trajectories is preferable, are much easier for users, and have emerged as an effective alternative. Unfortunately, comparisons are far less informative. We propose that there is much richer information that users can easily provide and that robots ought to leverage. We focus on augmenting comparisons with feature queries, and introduce a unified formalism for treating all answers as observations about the true desired reward. We derive an active query selection algorithm, and test these queries in simulation and on real users. We find that richer, feature-augmented queries can extract more information faster, leading to robots that better match user preferences in their behavior.