mlsanityData to 7 Oct 2026

ARCap: Collecting High-quality Human Demonstrations for Robot Learning with Augmented Reality Feedback

arXiv 2410.08464 ↗First posted 11 Oct 2024cs.RO cs.AIcited by 31

On the map: Offline RL and imitation learning from data → Robot manipulation imitation learning

Abstract (arXiv)

Recent progress in imitation learning from human demonstrations has shown promising results in teaching robots manipulation skills. To further scale up training datasets, recent works start to use portable data collection devices without the need for physical robot hardware. However, due to the absence of on-robot feedback during data collection, the data quality depends heavily on user expertise, and many devices are limited to specific robot embodiments. We propose ARCap, a portable data collection system that provides visual feedback through augmented reality (AR) and haptic warnings to guide users in collecting high-quality demonstrations. Through extensive user studies, we show that ARCap enables novice users to collect robot-executable data that matches robot kinematics and avoids collisions with the scenes. With data collected from ARCap, robots can perform challenging tasks, such as manipulation in cluttered environments and long-horizon cross-embodiment manipulation. ARCap is fully open-source and easy to calibrate; all components are built from off-the-shelf products. More details and results can be found on our website: https://stanford-tml.github.io/ARCap

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 28 of its 46 references that are in the map. First posted on arXiv 11 Oct 2024.

Builds on

28 of 46 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024Bunny-VisionPro: Real-Time Bimanual Dexterous Teleoperation for Imitation Learning3 Jul
  2. 2024Open-TeleVision: Teleoperation with Immersive Active Visual Feedback1 Jul
  3. 2024DexCap: Scalable and Portable Mocap Data Collection System for Dexterous Manipulation12 Mar
  4. 20243D Diffusion Policy: Generalizable Visuomotor Policy Learning via Simple 3D Representations6 Mar
  5. 2024Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperation4 Jan
  6. 2023GELLO: A General, Low-Cost, and Intuitive Teleoperation Framework for Robot Manipulators22 Sep
  7. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware23 Apr
  8. 2022VIOLA: Imitation Learning for Vision-Based Manipulation with Object Proposal Priors20 Oct
  9. 2021Generalization Through Hand-Eye Coordination: An Action Space for Learning Spatially-Invariant Visuomotor Control28 Feb
  10. 2019Self-Supervised Correspondence in Visuomotor Policy Learning16 Sep

Other references · 18

  1. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  2. 2024DROID: A Large-Scale In-The-Wild Robot Manipulation Datasetcited by 801
  3. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
  4. 2024Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robotscited by 373
  5. 2017One-Shot Visual Imitation Learning via Meta-Learningcited by 225
  6. 2024OmniH2O: Universal and Dexterous Human-to-Humanoid Whole-Body Teleoperation and Learningcited by 167
  7. 2024Consistency Policy: Accelerated Visuomotor Policies via Consistency Distillationcited by 159
  8. 2023AnyTeleop: A General Vision-Based Dexterous Robot Arm-Hand Teleoperation Systemcited by 136
  9. 2022VideoDex: Learning Dexterity from Internet Videoscited by 101
  10. 2023On Bringing Robots Homecited by 101
  11. 2024Learning Visuotactile Skills with Two Multifingered Handscited by 71
  12. 2023AirExo: Low-Cost Exoskeletons for Learning Whole-Arm Manipulation in the Wildcited by 46
6 more
  1. 2023Stabilize to Act: Learning to Coordinate for Bimanual Manipulationcited by 36
  2. 2024Efficient Data Collection for Robotic Manipulation via Compositional Generalizationcited by 35
  3. 2024ACE: A Cross-Platform Visual-Exoskeletons System for Low-Cost Dexterous Teleoperationcited by 32
  4. 2023AR2-D2:Training a Robot Without a Robotcited by 26
  5. 2020Telemanipulation with Chopsticks: Analyzing Human Factors in User Demonstrationscited by 5
  6. 2024Puppeteer Your Robot: Augmented Reality Leader-Follower Teleoperationcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 16; 2026: 12.

31 papers.

Sort:

By area:

  1. 2025EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Videocited by 191
  2. 2024ExBody2: Advanced Expressive Humanoid Whole-Body Controlcited by 75
  3. 2025ManipTrans: Efficient Dexterous Bimanual Manipulation Transfer via Residual Learningcited by 56
  4. 2025BEHAVIOR Robot Suite: Streamlining Real-World Whole-Body Manipulation for Everyday Household Activitiescited by 29
  5. 2026RDT2: Exploring the Scaling Limit of UMI Data Towards Zero-Shot Cross-Embodiment Generalizationcited by 29
  6. 2025Crossing the Human-Robot Embodiment Gap with Sim-to-Real RL using One Human Demonstrationcited by 27
  7. 2024Bridging the Human to Robot Dexterity Gap through Object-Oriented Rewardscited by 25
  8. 2025Physics-Driven Data Generation for Contact-Rich Manipulation via Trajectory Optimizationcited by 20
  9. 2025AirExo-2: Scaling up Generalizable Robotic Imitation Learning with Low-Cost Exoskeletonscited by 17
  10. 2025DemoDiffusion: One-Shot Human Imitation using pre-trained Diffusion Policycited by 16
  11. 2025Dexterity from Smart Lenses: Multi-Fingered Robot Manipulation with In-the-Wild Human Demonstrationscited by 14
  12. 2025Dexplore: Scalable Neural Control for Dexterous Manipulation from Reference-Scoped Explorationcited by 9
19 more
  1. 2025Human2LocoMan: Learning Versatile Quadrupedal Manipulation with Human Pretrainingcited by 8
  2. 2025Real-Time Operator Takeover for Visuomotor Diffusion Policy Trainingcited by 5
  3. 2026Lucid-XR: An Extended-Reality Data Engine for Robotic Manipulationcited by 4
  4. 2026RoboPocket: Improve Robot Policies Instantly with Your Phonecited by 3
  5. 2025IRIS: An Immersive Robot Interaction Systemcited by 2
  6. 2025Counterfactual Behavior Cloning: Offline Imitation Learning from Imperfect Human Demonstrationscited by 2
  7. 2025DeLTa: Demonstration and Language-Guided Novel Transparent Object Manipulationcited by 2
  8. 2026Ego-Pi: VLA Fine-Tuning for Ego-Centric Human and Robot Datacited by 2
  9. 2026YUBI: Yielding Universal Bidigital Interface for Bimanual Dexterous Manipulation at Scalecited by 2
  10. 2026HiFi-UMI: Learning Deployable Manipulation Policies from High-Fidelity UMI Data Alonecited by 2
  11. 2025Learning human-to-robot handovers through 3D scene reconstructioncited by 1
  12. 2025SFHand: Learning Embodied Manipulation by Streaming Egocentric 3D Hand Forecastingcited by 1
  13. 2026Revisiting Articulated Parts Perception in Robot Manipulationcited by 1
  14. 2024RAMPA: Robotic Augmented Reality for Machine Programming by DemonstrAtioncited by 0
  15. 2026Feasibility-aware Imitation Learning from Observation with Multimodal Feedbackcited by 0
  16. 2026COBALT: Crowdsourcing Robot Learning via Cloud-Based Teleoperation with Smartphonescited by 0
  17. 2026Tac2Pix: Image-Space Visuo-Tactile Fusion for Dexterous Manipulationcited by 0
  18. 2026AdvDex: Learning Dexterous Manipulation from Human Demonstrations via Joint-Aligned Actions and Adversarial Learningcited by 0
  19. 2026Touch2Robot: Robot Touch in the Human Demonstration Loopcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Augmented Reality Demonstrations for Scalable Robot Imitation Learning
  2. 2024ARMADA: Augmented Reality for Robot Manipulation and Robot-Free Data Acquisition
  3. 2025ARMimic: Learning Robotic Manipulation from Passive Human Demonstrations in Augmented Reality
  4. 2024ARCADE: Scalable Demonstration Collection and Generation via Augmented Reality for Imitation Learning
  5. 2025RoboManipBaselines: A Unified Framework for Imitation Learning in Robotic Manipulation across Real and Simulation Environments
  6. 2020Visual Imitation Made Easy
  7. 2022Leveraging Haptic Feedback to Improve Data Quality and Quantity for Deep Imitation Learning Models
  8. 2025MV-UMI: A Scalable Multi-View Interface for Cross-Embodiment Learning