mlsanityData to 14 Oct 2026

Learning to Discern: Imitating Heterogeneous Human Demonstrations with Preference and Representation Learning

arXiv 2310.14196 ↗First posted 22 Oct 2023cs.RO cs.AIcited by 15

On the map: Offline RL and imitation learning from data → Robot manipulation imitation learning

Abstract (arXiv)

Practical Imitation Learning (IL) systems rely on large human demonstration datasets for successful policy learning. However, challenges lie in maintaining the quality of collected data and addressing the suboptimal nature of some demonstrations, which can compromise the overall dataset quality and hence the learning outcome. Furthermore, the intrinsic heterogeneity in human behavior can produce equally successful but disparate demonstrations, further exacerbating the challenge of discerning demonstration quality. To address these challenges, this paper introduces Learning to Discern (L2D), an offline imitation learning framework for learning from demonstrations with diverse quality and style. Given a small batch of demonstrations with sparse quality labels, we learn a latent representation for temporally embedded trajectory segments. Preference learning in this latent space trains a quality evaluator that generalizes to new demonstrators exhibiting different styles. Empirically, we show that L2D can effectively assess and learn from varying demonstrations, thereby leading to improved policy performance across a range of tasks in both simulations and on a physical robot.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 28 of its 41 references that are in the map. First posted on arXiv 22 Oct 2023.

Builds on

28 of 41 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023Behavior Retrieval: Few-Shot Imitation Learning by Querying Unlabeled Datasets18 Apr
  2. 2022Eliciting Compatible Demonstrations for Multi-Human Imitation Learning14 Oct
  3. 2022Behavior Transformers: Cloning k modes with one stone22 Jun
  4. 2020Learning to Generalize Across Long-Horizon Tasks from Human Demonstrations13 Mar
  5. 2018RoboTurk: A Crowdsourcing Platform for Robotic Skill Learning through Imitation7 Nov

Other references · 23

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2017Deep reinforcement learning from human preferencescited by 3,912
  3. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,838
  4. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
  5. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  6. 2016Generative Adversarial Imitation Learningcited by 1,597
  7. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 725
  8. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 694
  9. 2020Reinforcement Learning with Augmented Datacited by 468
  10. 2021Implicit Behavioral Cloningcited by 452
  11. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 406
  12. 2018Reward learning from human preferences and demonstrations in Ataricited by 316
11 more
  1. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 302
  2. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 260
  3. 2021RvS: What is Essential for Offline RL via Supervised Learning?cited by 185
  4. 2019Goal-conditioned Imitation Learningcited by 162
  5. 2022When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning?cited by 89
  6. 2020Learning from Suboptimal Demonstration via Self-Supervised Reward Regressioncited by 84
  7. 2023Preference Transformer: Modeling Human Preferences using Transformers for RLcited by 79
  8. 2022Offline RL Policies Should be Trained to be Adaptivecited by 52
  9. 2022Imitation Learning by Estimating Expertise of Demonstratorscited by 50
  10. 2020Goal-Aware Prediction: Learning to Model What Matterscited by 45
  11. 2021Confidence-Aware Imitation Learning from Demonstrations with Varying Optimalitycited by 44

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 6; 2026: 6.

15 papers.

Sort:

By area:

  1. 2025CUPID: Curating Data your Robot Loves with Influence Functionscited by 23
  2. 2025Curating Demonstrations using Online Experiencecited by 18
  3. 2024Offline Imitation Learning Through Graph Search and Retrievalcited by 14
  4. 2024Inverse Reinforcement Learning by Estimating Expertise of Demonstratorscited by 11
  5. 2025Data Retrieval with Importance Weights for Few-Shot Imitation Learningcited by 10
  6. 2025Counterfactual Behavior Cloning: Offline Imitation Learning from Imperfect Human Demonstrationscited by 3
  7. 2025Demonstration Sidetracks: Categorizing Systematic Non-Optimality in Human Demonstrationscited by 2
  8. 2025Similarity as Reward Alignment: Robust and Versatile Preference-based Reinforcement Learningcited by 2
  9. 2024Multi-Stage Multi-Modal Pre-Training for Automatic Speech Recognitioncited by 1
  10. 2026ATHENA: Accelerated Multi-Task Heterogeneous Influence Functions for Robot Data Curationcited by 1
  11. 2026Behavior Uncloning: Distilling Mode Redirection into Policy Weights without Inference-Time Steeringcited by 1
  12. 2026TSD: A Physics-Inspired Trajectory Saliency Detector for Efficient Imitation Learningcited by 0
3 more
  1. 2026Imperfection for Precision: Upcycling Imperfect Data for High-Precision Robotic Manipulationcited by 0
  2. 2026Learning from Mixed-Quality Deployment Experience for Robot Manipulationcited by 0
  3. 2026Where Small Mistakes Matter: Quantifying State Criticality for Imitation Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Robust Offline Imitation Learning Through State-level Trajectory Stitching
  2. 2024Towards Diverse Behaviors: A Benchmark for Imitation Learning with Human Demonstrations
  3. 2025Robot Data Curation with Mutual Information Estimators
  4. 2026An Efficient Metric for Data Quality Measurement in Imitation Learning
  5. 2024A Dual Approach to Imitation Learning from Observations with Offline Datasets
  6. 2026Difference-Aware Retrieval Policies for Imitation Learning
  7. 2019VILD: Variational Imitation Learning with Diverse-quality Demonstrations
  8. 2021Learning from Imperfect Demonstrations from Agents with Varying Dynamics