# Learning to Discern: Imitating Heterogeneous Human Demonstrations with Preference and Representation Learning

arXiv: https://arxiv.org/abs/2310.14196 · HTML: https://mlsanity.com/paper/2310.14196/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 28 of its 41 references that are in the map.
- First posted on arXiv 22 Oct 2023.

## Placed in

- Offline RL and imitation learning from data / [Robot manipulation imitation learning](https://mlsanity.com/offlinerl/robot-manipulation-imitation-learning/)

## Builds on (most cited first, 28 of 28)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [RT-1: Robotics Transformer for Real-World Control at Scale](https://mlsanity.com/paper/2212.06817/) (2022)
- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [What Matters in Learning from Offline Human Demonstrations for Robot Manipulation](https://mlsanity.com/paper/2108.03298/) (2021)
- [R3M: A Universal Visual Representation for Robot Manipulation](https://mlsanity.com/paper/2203.12601/) (2022)
- [Reinforcement Learning with Augmented Data](https://mlsanity.com/paper/2004.14990/) (2020)
- [Implicit Behavioral Cloning](https://mlsanity.com/paper/2109.00137/) (2021)
- [Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization](https://mlsanity.com/paper/1603.00448/) (2016)
- [Reward learning from human preferences and demonstrations in Atari](https://mlsanity.com/paper/1811.06521/) (2018)
- [Behavior Transformers: Cloning k modes with one stone](https://mlsanity.com/paper/2206.11251/) (2022)
- [PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training](https://mlsanity.com/paper/2106.05091/) (2021)
- [Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observations](https://mlsanity.com/paper/1904.06387/) (2019)
- [RoboTurk: A Crowdsourcing Platform for Robotic Skill Learning through Imitation](https://mlsanity.com/paper/1811.02790/) (2018)
- [RvS: What is Essential for Offline RL via Supervised Learning?](https://mlsanity.com/paper/2112.10751/) (2021)
- [Goal-conditioned Imitation Learning](https://mlsanity.com/paper/1906.05838/) (2019)
- [Learning to Generalize Across Long-Horizon Tasks from Human Demonstrations](https://mlsanity.com/paper/2003.06085/) (2020)
- [When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning?](https://mlsanity.com/paper/2204.05618/) (2022)
- [Learning from Suboptimal Demonstration via Self-Supervised Reward Regression](https://mlsanity.com/paper/2010.11723/) (2020)
- [Preference Transformer: Modeling Human Preferences using Transformers for RL](https://mlsanity.com/paper/2303.00957/) (2023)
- [Behavior Retrieval: Few-Shot Imitation Learning by Querying Unlabeled Datasets](https://mlsanity.com/paper/2304.08742/) (2023)
- [Offline RL Policies Should be Trained to be Adaptive](https://mlsanity.com/paper/2207.02200/) (2022)
- [Imitation Learning by Estimating Expertise of Demonstrators](https://mlsanity.com/paper/2202.01288/) (2022)
- [Goal-Aware Prediction: Learning to Model What Matters](https://mlsanity.com/paper/2007.07170/) (2020)
- [Confidence-Aware Imitation Learning from Demonstrations with Varying Optimality](https://mlsanity.com/paper/2110.14754/) (2021)
- [Eliciting Compatible Demonstrations for Multi-Human Imitation Learning](https://mlsanity.com/paper/2210.08073/) (2022)

## Built on by (most cited first, 15 of 15)

- [CUPID: Curating Data your Robot Loves with Influence Functions](https://mlsanity.com/paper/2506.19121/) (2025)
- [Curating Demonstrations using Online Experience](https://mlsanity.com/paper/2503.03707/) (2025)
- [Offline Imitation Learning Through Graph Search and Retrieval](https://mlsanity.com/paper/2407.15403/) (2024)
- [Inverse Reinforcement Learning by Estimating Expertise of Demonstrators](https://mlsanity.com/paper/2402.01886/) (2024)
- [Data Retrieval with Importance Weights for Few-Shot Imitation Learning](https://mlsanity.com/paper/2509.01657/) (2025)
- [Counterfactual Behavior Cloning: Offline Imitation Learning from Imperfect Human Demonstrations](https://mlsanity.com/paper/2505.10760/) (2025)
- [Demonstration Sidetracks: Categorizing Systematic Non-Optimality in Human Demonstrations](https://mlsanity.com/paper/2506.11262/) (2025)
- [Similarity as Reward Alignment: Robust and Versatile Preference-based Reinforcement Learning](https://mlsanity.com/paper/2506.12529/) (2025)
- [Multi-Stage Multi-Modal Pre-Training for Automatic Speech Recognition](https://mlsanity.com/paper/2403.19822/) (2024)
- [ATHENA: Accelerated Multi-Task Heterogeneous Influence Functions for Robot Data Curation](https://mlsanity.com/paper/2606.16208/) (2026)
- [Behavior Uncloning: Distilling Mode Redirection into Policy Weights without Inference-Time Steering](https://mlsanity.com/paper/2606.29201/) (2026)
- [TSD: A Physics-Inspired Trajectory Saliency Detector for Efficient Imitation Learning](https://mlsanity.com/paper/2606.23371/) (2026)
- [Imperfection for Precision: Upcycling Imperfect Data for High-Precision Robotic Manipulation](https://mlsanity.com/paper/2609.26672/) (2026)
- [Learning from Mixed-Quality Deployment Experience for Robot Manipulation](https://mlsanity.com/paper/2609.29000/) (2026)
- [Where Small Mistakes Matter: Quantifying State Criticality for Imitation Learning](https://mlsanity.com/paper/2610.17709/) (2026)

## Abstract (arXiv)

Practical Imitation Learning (IL) systems rely on large human demonstration datasets for successful policy learning. However, challenges lie in maintaining the quality of collected data and addressing the suboptimal nature of some demonstrations, which can compromise the overall dataset quality and hence the learning outcome. Furthermore, the intrinsic heterogeneity in human behavior can produce equally successful but disparate demonstrations, further exacerbating the challenge of discerning demonstration quality. To address these challenges, this paper introduces Learning to Discern (L2D), an offline imitation learning framework for learning from demonstrations with diverse quality and style. Given a small batch of demonstrations with sparse quality labels, we learn a latent representation for temporally embedded trajectory segments. Preference learning in this latent space trains a quality evaluator that generalizes to new demonstrators exhibiting different styles. Empirically, we show that L2D can effectively assess and learn from varying demonstrations, thereby leading to improved policy performance across a range of tasks in both simulations and on a physical robot.