mlsanityData to 7 Oct 2026

Human-in-the-Loop Task and Motion Planning for Imitation Learning

arXiv 2310.16014 ↗First posted 24 Oct 2023cs.RO cs.AI cs.CVcited by 28

On the map: Offline RL and imitation learning from data → Robot manipulation imitation learning

Abstract (arXiv)

Imitation learning from human demonstrations can teach robots complex manipulation skills, but is time-consuming and labor intensive. In contrast, Task and Motion Planning (TAMP) systems are automated and excel at solving long-horizon tasks, but they are difficult to apply to contact-rich tasks. In this paper, we present Human-in-the-Loop Task and Motion Planning (HITL-TAMP), a novel system that leverages the benefits of both approaches. The system employs a TAMP-gated control mechanism, which selectively gives and takes control to and from a human teleoperator. This enables the human teleoperator to manage a fleet of robots, maximizing data collection efficiency. The collected human data is then combined with an imitation learning framework to train a TAMP-gated policy, leading to superior performance compared to training on full task demonstrations. We compared HITL-TAMP to a conventional teleoperation system -- users gathered more than 3x the number of demos given the same time budget. Furthermore, proficient agents (75%+ success) could be trained from just 10 minutes of non-expert teleoperation data. Finally, we collected 2.1K demos with HITL-TAMP across 12 contact-rich, long-horizon tasks and show that the system often produces near-perfect agents. Videos and additional results at https://hitltamp.github.io .

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 46 of its 71 references that are in the map. First posted on arXiv 24 Oct 2023.

Builds on

46 of 71 references.

Closest ancestors · 9

Same direction, newest first

  1. 2023MimicPlay: Long-Horizon Imitation Learning by Watching Human Play24 Feb
  2. 2022VIOLA: Imitation Learning for Vision-Based Manipulation with Object Proposal Priors20 Oct
  3. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning4 Feb
  4. 2021LazyDAgger: Reducing Context Switching in Interactive Imitation Learning31 Mar
  5. 2021Generalization Through Hand-Eye Coordination: An Action Space for Learning Spatially-Invariant Visuomotor Control28 Feb
  6. 2020Human-in-the-Loop Imitation Learning using Remote Teleoperation12 Dec
  7. 2020Learning to Generalize Across Long-Horizon Tasks from Human Demonstrations13 Mar
  8. 2018RoboTurk: A Crowdsourcing Platform for Robotic Skill Learning through Imitation7 Nov
  9. 2018Reinforcement and Imitation Learning for Diverse Visuomotor Skills26 Feb

Other references · 37

  1. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  2. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  3. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
  4. 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 564
  5. 2020robosuite: A Modular Simulation Framework and Benchmark for Robot Learningcited by 484
  6. 2019Learning Latent Plans from Playcited by 310
  7. 2017Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperationcited by 286
  8. 2020Integrated Task and Motion Planningcited by 281
  9. 2018HG-DAgger: Interactive Imitation Learning with Human Expertscited by 245
  10. 2018Residual Reinforcement Learning for Robot Controlcited by 242
  11. 2022Interactive Language: Talking to Robots in Real Timecited by 217
  12. 2021Contact-GraspNet: Efficient 6-DoF Grasp Generation in Cluttered Scenescited by 213
25 more
  1. 2018PDDLStream: Integrating Symbolic Planners and Blackbox Samplers via Optimistic Adaptive Planningcited by 147
  2. 2018Residual Policy Learningcited by 146
  3. 2022Grounding Language with Visual Affordances over Unstructured Datacited by 101
  4. 2021ThriftyDAgger: Budget-Aware Novelty and Risk Gating for Interactive Imitation Learningcited by 98
  5. 2018EnsembleDAgger: A Bayesian Approach to Safe Imitation Learningcited by 96
  6. 2018Hierarchical Imitation and Reinforcement Learningcited by 88
  7. 2019IRIS: Implicit Reinforcement without Interaction at Scale for Learning Control from Offline Robot Manipulation Datacited by 85
  8. 2016Query-Efficient Imitation Learning for End-to-End Autonomous Drivingcited by 83
  9. 2018TACO: Learning Task Decomposition via Temporal Alignment for Controlcited by 61
  10. 2021Learning Symbolic Operators for Task and Motion Planningcited by 57
  11. 2022Learning Neuro-Symbolic Skills for Bilevel Planningcited by 55
  12. 2020Learning compositional models of robot skills for task and motion planningcited by 47
  13. 2022Efficient Learning of Safe Driving Policy via Human-AI Copilot Optimizationcited by 44
  14. 2021Long-Horizon Manipulation of Unknown Objects via Task and Motion Planning with Estimated Affordancescited by 38
  15. 2022LEAGUE: Guided Skill Learning and Abstraction for Long-Horizon Manipulationcited by 32
  16. 2022Fleet-DAgger: Interactive Robot Fleet Learning with Scalable Human Supervisioncited by 29
  17. 2019AC-Teach: A Bayesian Actor-Critic Method for Policy Learning with an Ensemble of Suboptimal Teacherscited by 27
  18. 2021Search-Based Task Planning with Learned Skill Effect Models for Lifelong Robotic Manipulationcited by 20
  19. 2022PATO: Policy Assisted TeleOperation for Scalable Robot Data Collectioncited by 19
  20. 2021Vision-driven Compliant Manipulation for Reliable, High-Precision Assembly Taskscited by 18
  21. 2021Guided Imitation of Task and Motion Planningcited by 18
  22. 2021Co-GAIL: Learning Diverse Strategies for Human-Robot Collaborationcited by 16
  23. 2021Coarse-to-Fine for Sim-to-Real: Sub-Millimetre Precision Across Wide Task Spacescited by 9
  24. 2022Representation, learning, and planning algorithms for geometric task and motion planningcited by 8
  25. 2021Learning to Share Autonomy Across Repeated Interactioncited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 6; 2025: 11; 2026: 10.

28 papers.

Sort:

By area:

  1. 2024Plan-Seq-Learn: Language Model Guided RL for Solving Long Horizon Robotics Taskscited by 51
  2. 2024SkillMimicGen: Automated Demonstration Generation for Efficient Skill Learning and Deploymentcited by 48
  3. 2024Don't Start from Scratch: Behavioral Refinement via Interpolant-based Policy Diffusioncited by 21
  4. 2024A Survey of Optimization-based Task and Motion Planning: From Classical To Learning Approachescited by 19
  5. 2025Learning Compositional Behaviors from Demonstration and Languagecited by 15
  6. 2026LiLo-VLA: Compositional Long-Horizon Manipulation via Linked Object-Centric Policiescited by 12
  7. 2023NOD-TAMP: Generalizable Long-Horizon Planning with Neural Object Descriptorscited by 8
  8. 2024SPIRE: Synergistic Planning, Imitation, and Reinforcement Learning for Long-Horizon Manipulationcited by 8
  9. 2025Hybrid Diffusion for Simultaneous Symbolic and Continuous Planningcited by 6
  10. 2025Neuro-Symbolic Imitation Learning: Discovering Symbolic Abstractions for Skill Learningcited by 4
  11. 2026FAR: Failure-Aware Retry for Test-Time Recovery and Continual Policy Improvementcited by 4
  12. 2025HybridGen: VLM-Guided Hybrid Planning for Scalable Data Generation of Imitation Learningcited by 3
16 more
  1. 2025Interactive Imitation Learning for Dexterous Robotic Manipulation: Challenges and Perspectives -- A Surveycited by 3
  2. 2025Advancing Responsible Innovation in Agentic AI: A study of Ethical Frameworks for Household Automationcited by 3
  3. 2025SLAP: Shortcut Learning for Abstract Planningcited by 3
  4. 2025ReinforceGen: Hybrid Skill Policies with Automated Data Generation and Reinforcement Learningcited by 3
  5. 2025IRIS: An Immersive Robot Interaction Systemcited by 2
  6. 2026FAR-Dex: Few-shot Data Augmentation and Adaptive Residual Policy Refinement for Dexterous Manipulationcited by 2
  7. 2024Logic Learning from Demonstrations for Multi-step Manipulation Tasks in Dynamic Environmentscited by 1
  8. 2026HumanoidMimicGen: Data Generation for Loco-Manipulation via Whole-Body Planningcited by 1
  9. 2026DREAM: Deployment-Time Demonstration Generation via Real-to-Sim for Scalable Policy Adaptationcited by 1
  10. 2026DATAFARM: Distribution-Aligned Task and Motion Planning for Fine-Tuning Vision-Language-Action Modelscited by 1
  11. 2025Query-Centric Diffusion Policy for Generalizable Robotic Assemblycited by 0
  12. 2025Learning to Plan & Schedule with Reinforcement-Learned Bimanual Robot Skillscited by 0
  13. 2026Learning Bilevel Policies over Symbolic World Models for Long-Horizon Planningcited by 0
  14. 2026Simulation-Driven Imitation Learning for Biosignals-Free Shared-Autonomy Prosthetic Graspingcited by 0
  15. 2026Human2Any: Human-to-Robot Transfer via Constraint-Aware Compositional Planningcited by 0
  16. 2026TANDEM: Task and Motion Planning with As-Needed Demonstrations for Efficient Vision-Language-Action Model Fine-tuningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Imitating Task and Motion Planning with Visuomotor Transformers
  2. 2026Human-Robot Copilot for Data-Efficient Imitation Learning
  3. 2025Hybrid Imitation Learning: Teleoperation Augmentation Primitives that Policies Learn to Trigger
  4. 2020Learning Multi-Arm Manipulation Through Collaborative Teleoperation
  5. 2025RoboCopilot: Human-in-the-loop Interactive Imitation Learning for Robot Manipulation
  6. 2024VITAL: Interactive Few-Shot Imitation Learning via Visual Human-in-the-Loop Corrections
  7. 2023GELLO: A General, Low-Cost, and Intuitive Teleoperation Framework for Robot Manipulators
  8. 2026Decompose and Reorganize: Planning with Primitives and Visuomotor Policies Learned from Demonstrations ↗