# FARPLS: A Feature-Augmented Robot Trajectory Preference Labeling System to Assist Human Labelers' Preference Elicitation

arXiv: https://arxiv.org/abs/2403.06267 · HTML: https://mlsanity.com/paper/2403.06267/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 93 references that are in the map.
- First posted on arXiv 10 Mar 2024.

## Builds on (most cited first, 21 of 21)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2307.15217/) (2023)
- [What Matters in Learning from Offline Human Demonstrations for Robot Manipulation](https://mlsanity.com/paper/2108.03298/) (2021)
- [Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observations](https://mlsanity.com/paper/1904.06387/) (2019)
- [Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons](https://mlsanity.com/paper/2301.11270/) (2023)
- [The VIA Annotation Software for Images, Audio and Video](https://mlsanity.com/paper/1904.10699/) (2019)
- [Reinforcement Learning with Human Feedback: Learning Dynamic Choices via Pessimism](https://mlsanity.com/paper/2305.18438/) (2023)
- [Asking Easy Questions: A User-Friendly Approach to Active Reward Learning](https://mlsanity.com/paper/1910.04365/) (2019)
- [Benchmarks and Algorithms for Offline Preference-Based Reward Learning](https://mlsanity.com/paper/2301.01392/) (2023)
- [Feature Expansive Reward Learning: Rethinking Human Input](https://mlsanity.com/paper/2006.13208/) (2020)
- [Learning Reward Functions from Scale Feedback](https://mlsanity.com/paper/2110.00284/) (2021)
- [Perspectives on the Social Impacts of Reinforcement Learning with Human Feedback](https://mlsanity.com/paper/2303.02891/) (2023)
- [Eliciting Single-Peaked Preferences Using Comparison Queries](https://mlsanity.com/paper/1401.3449/) (2014)
- [Human-guided Robot Behavior Learning: A GAN-assisted Preference-based Reinforcement Learning Approach](https://mlsanity.com/paper/2010.07467/) (2020)
- [OneLabeler: A Flexible System for Building Data Labeling Tools](https://mlsanity.com/paper/2203.14227/) (2022)
- [Semi-Automatic Labeling for Deep Learning in Robotics](https://mlsanity.com/paper/1908.01862/) (2019)
- [Annotation Curricula to Implicitly Train Non-Expert Annotators](https://mlsanity.com/paper/2106.02382/) (2021)
- [Here's What I've Learned: Asking Questions that Reveal Reward Learning](https://mlsanity.com/paper/2107.01995/) (2021)
- [MyMove: Facilitating Older Adults to Collect In-Situ Activity Labels on a Smartwatch with Speech](https://mlsanity.com/paper/2204.00145/) (2022)
- [A User Interface for Sense-making of the Reasoning Process while Interacting with Robots](https://mlsanity.com/paper/2210.08246/) (2022)

## Built on by (most cited first, 1 of 1)

- [Towards Considerate Embodied AI: Co-Designing Situated Multi-Site Healthcare Robots from Abstract Concepts to High-Fidelity Prototypes](https://mlsanity.com/paper/2602.03054/) (2026)

## Abstract (arXiv)

Preference-based learning aims to align robot task objectives with human values. One of the most common methods to infer human preferences is by pairwise comparisons of robot task trajectories. Traditional comparison-based preference labeling systems seldom support labelers to digest and identify critical differences between complex trajectories recorded in videos. Our formative study (N = 12) suggests that individuals may overlook non-salient task features and establish biased preference criteria during their preference elicitation process because of partial observations. In addition, they may experience mental fatigue when given many pairs to compare, causing their label quality to deteriorate. To mitigate these issues, we propose FARPLS, a Feature-Augmented Robot trajectory Preference Labeling System. FARPLS highlights potential outliers in a wide variety of task features that matter to humans and extracts the corresponding video keyframes for easy review and comparison. It also dynamically adjusts the labeling order according to users' familiarities, difficulties of the trajectory pair, and level of disagreements. At the same time, the system monitors labelers' consistency and provides feedback on labeling progress to keep labelers engaged. A between-subjects study (N = 42, 105 pairs of robot pick-and-place trajectories per person) shows that FARPLS can help users establish preference criteria more easily and notice more relevant details in the presented trajectories than the conventional interface. FARPLS also improves labeling consistency and engagement, mitigating challenges in preference elicitation without raising cognitive loads significantly