mlsanityData to 7 Oct 2026

PATO: Policy Assisted TeleOperation for Scalable Robot Data Collection

arXiv 2212.04708 ↗First posted 9 Dec 2022cs.RO cs.AI cs.LGcited by 19

Not on the map. Why?

Abstract (arXiv)

Large-scale data is an essential component of machine learning as demonstrated in recent advances in natural language processing and computer vision research. However, collecting large-scale robotic data is much more expensive and slower as each operator can control only a single robot at a time. To make this costly data collection process efficient and scalable, we propose Policy Assisted TeleOperation (PATO), a system which automates part of the demonstration collection process using a learned assistive policy. PATO autonomously executes repetitive behaviors in data collection and asks for human input only when it is uncertain about which subtask or behavior to execute. We conduct teleoperation user studies both with a real robot and a simulated robot fleet and demonstrate that our assisted teleoperation system reduces human operators' mental load while improving data collection efficiency. Further, it enables a single operator to control multiple robots in parallel, which is a first step towards scalable robotic data collection. For code and video results, see https://clvrai.com/pato

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 27 of its 53 references that are in the map. First posted on arXiv 9 Dec 2022.

Builds on

27 of 53 references.

References · 27

  1. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  2. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
  3. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  4. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learningcited by 380
  5. 2019Learning Latent Plans from Playcited by 310
  6. 2017Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperationcited by 286
  7. 2021Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasetscited by 248
  8. 2018HG-DAgger: Interactive Imitation Learning with Human Expertscited by 245
  9. 2018RoboTurk: A Crowdsourcing Platform for Robotic Skill Learning through Imitationcited by 202
  10. 2018Efficient Interactive Annotation of Segmentation Datasets with Polygon-RNN++cited by 119
  11. 2020Learning to Generalize Across Long-Horizon Tasks from Human Demonstrationscited by 114
  12. 2021ThriftyDAgger: Budget-Aware Novelty and Risk Gating for Interactive Imitation Learningcited by 98
15 more
  1. 2018EnsembleDAgger: A Bayesian Approach to Safe Imitation Learningcited by 96
  2. 2017Annotating Object Instances with a Polygon-RNNcited by 88
  3. 2019IRIS: Implicit Reinforcement without Interaction at Scale for Learning Control from Offline Robot Manipulation Datacited by 85
  4. 2016Query-Efficient Imitation Learning for End-to-End Autonomous Drivingcited by 83
  5. 2019IKEA Furniture Assembly Environment for Long-Horizon Complex Manipulation Taskscited by 68
  6. 2021LazyDAgger: Reducing Context Switching in Interactive Imitation Learningcited by 50
  7. 2015Shared Autonomy via Hindsight Optimizationcited by 46
  8. 2021AW-Opt: Learning Robotic Skills with Imitation and Reinforcement at Scalecited by 43
  9. 2020Shared Autonomy with Learned Latent Actionscited by 34
  10. 2019Controlling Assistive Robots with Learned Latent Actionscited by 28
  11. 2020A Quality Diversity Approach to Automatically Generating Human-Robot Interaction Scenarios in Shared Autonomycited by 20
  12. 2017Human-Robot Mutual Adaptation in Shared Autonomycited by 16
  13. 2020Four Years in Review: Statistical Practices of Likert Scales in Human-Robot Interaction Studiescited by 12
  14. 2021Assistive Tele-op: Leveraging Transformers to Collect Robotic Task Demonstrationscited by 11
  15. 2021A System for Traded Control Teleoperation of Manipulation Tasks using Intent Prediction from Hand Gesturescited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 7; 2024: 3; 2025: 5; 2026: 4.

19 papers.

Sort:

By area:

  1. 2023MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrationscited by 308
  2. 2023Imitating Task and Motion Planning with Visuomotor Transformerscited by 63
  3. 2023Model-Based Runtime Monitoring with Interactive Imitation Learningcited by 37
  4. 2025Dual-Stream Diffusion for World-Model Augmented Vision-Language-Action Modelcited by 33
  5. 2024Diff-DAgger: Uncertainty Estimation with Diffusion Policy for Robotic Manipulationcited by 29
  6. 2023Human-in-the-Loop Task and Motion Planning for Imitation Learningcited by 28
  7. 2023Interactive Robot Learning from Verbal Correctioncited by 28
  8. 2024TeleMoMa: A Modular and Versatile Teleoperation System for Mobile Manipulationcited by 27
  9. 2025DataMIL: Selecting Data for Robot Imitation Learning with Datamodelscited by 25
  10. 2023SPRINT: Scalable Policy Pre-Training via Language Instruction Relabelingcited by 24
  11. 2025EveryDayVLA: A Vision-Language-Action Model for Affordable Robotic Manipulationcited by 5
  12. 2026VGAS: Value-Guided Action-Chunk Selection for Few-Shot Vision-Language-Action Adaptationcited by 5
7 more
  1. 2025FieldGen: From Teleoperated Pre-Manipulation Trajectories to Field-Guided Data Generationcited by 4
  2. 2025From Power to Precision: Learning Fine-grained Dexterity for Multi-fingered Robotic Handscited by 2
  3. 2023Modifying RL Policies with Imagined Actions: How Predictable Policies Can Enable Users to Perform Novel Taskscited by 1
  4. 2024Imagining In-distribution States: How Predictable Robot Behavior Can Enable User Control Over Learned Policiescited by 0
  5. 2026Assistron: Bayesian Shared Autonomy with Off-the-shelf Vision-Language-Action Modelscited by 0
  6. 2026HIL-UMI: Bringing Human-in-the-Loop Post-Training of Vision-Language-Action Models to Universal Manipulation Interfacecited by 0
  7. 2026TANDEM: Task and Motion Planning with As-Needed Demonstrations for Efficient Vision-Language-Action Model Fine-tuningcited by 0