# UMI-3D: Extending Universal Manipulation Interface from Vision-Limited to 3D Spatial Perception arXiv: https://arxiv.org/abs/2604.14089 · HTML: https://mlsanity.com/paper/2604.14089/ ## Key facts - Cited by 4 papers in the map (data through 14 Oct 2026). - Builds on 31 of its 40 references that are in the map. - First posted on arXiv 15 Apr 2026. ## Builds on (most cited first, 30 of 31) - [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021) - [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020) - [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023) - [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023) - [π0: A Vision-Language-Action Flow Model for General Robot Control](https://mlsanity.com/paper/2410.24164/) (2024) - [Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware](https://mlsanity.com/paper/2304.13705/) (2023) - [ORB-SLAM: a Versatile and Accurate Monocular SLAM System](https://mlsanity.com/paper/1502.00956/) (2015) - [ORB-SLAM3: An Accurate Open-Source Library for Visual, Visual-Inertial and Multi-Map SLAM](https://mlsanity.com/paper/2007.11898/) (2020) - [VINS-Mono: A Robust and Versatile Monocular Visual-Inertial State Estimator](https://mlsanity.com/paper/1708.03852/) (2017) - [Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots](https://mlsanity.com/paper/2402.10329/) (2024) - [LIO-SAM: Tightly-coupled Lidar Inertial Odometry via Smoothing and Mapping](https://mlsanity.com/paper/2007.00258/) (2020) - [FAST-LIO2: Fast Direct LiDAR-inertial Odometry](https://mlsanity.com/paper/2107.06829/) (2021) - [Data Scaling Laws in Imitation Learning for Robotic Manipulation](https://mlsanity.com/paper/2410.18647/) (2024) - [UMI on Legs: Making Manipulation Policies Mobile with Manipulation-Centric Whole-body Controllers](https://mlsanity.com/paper/2407.10353/) (2024) - [DexUMI: Using Human Hand as the Universal Manipulation Interface for Dexterous Manipulation](https://mlsanity.com/paper/2505.21864/) (2025) - [FastUMI: A Scalable and Hardware-Independent Universal Manipulation Interface with Dataset](https://mlsanity.com/paper/2409.19499/) (2024) - [RDT2: Exploring the Scaling Limit of UMI Data Towards Zero-Shot Cross-Embodiment Generalization](https://mlsanity.com/paper/2602.03310/) (2026) - [Automatic Extrinsic Calibration Method for LiDAR and Camera Sensor Setups](https://mlsanity.com/paper/2101.04431/) (2021) - [ActiveUMI: Robotic Manipulation with Active Perception from Robot-Free Human Demonstrations](https://mlsanity.com/paper/2510.01607/) (2025) - [Efficient and Probabilistic Adaptive Voxel Mapping for Accurate Online LiDAR Odometry](https://mlsanity.com/paper/2109.07082/) (2021) - [HoMMI: Learning Whole-Body Mobile Manipulation from Human Demonstrations](https://mlsanity.com/paper/2603.03243/) (2026) - [exUMI: Extensible Robot Teaching System with Action-aware Task-agnostic Tactile Representation](https://mlsanity.com/paper/2509.14688/) (2025) - [In-the-Wild Compliant Manipulation with UMI-FT](https://mlsanity.com/paper/2601.09988/) (2026) - [Low-cost Retina-like Robotic Lidars Based on Incommensurable Scanning](https://mlsanity.com/paper/2006.11034/) (2020) - [Kalman Filters on Differentiable Manifolds](https://mlsanity.com/paper/2102.03804/) (2021) - [RoboPocket: Improve Robot Policies Instantly with Your Phone](https://mlsanity.com/paper/2603.05504/) (2026) - [FAST-Calib: LiDAR-Camera Extrinsic Calibration in One Second](https://mlsanity.com/paper/2507.17210/) (2025) - [UMIGen: A Unified Framework for Egocentric Point Cloud Generation and Cross-Embodiment Robotic Imitation Learning](https://mlsanity.com/paper/2511.09302/) (2025) - [Rethinking Camera Choice: An Empirical Study on Fisheye Camera Properties in Robotic Manipulation](https://mlsanity.com/paper/2603.02139/) (2026) - [A Survey on LiDAR-based Autonomous Aerial Vehicles](https://mlsanity.com/paper/2509.10730/) (2025) ## Built on by (most cited first, 4 of 4) - [Data Pyramid for Embodied Manipulation: A Survey](https://mlsanity.com/paper/2607.24744/) (2026) - [YUBI: Yielding Universal Bidigital Interface for Bimanual Dexterous Manipulation at Scale](https://mlsanity.com/paper/2606.10244/) (2026) - [VISTA: Vision-Grounded and Physics-Validated Adaptation of UMI data for VLA Training](https://mlsanity.com/paper/2606.04708/) (2026) - [Pre-training Visual Dexterity in Simulation](https://mlsanity.com/paper/2608.15917/) (2026) ## Abstract (arXiv) We present UMI-3D, a multimodal extension of the Universal Manipulation Interface (UMI) for robust and scalable data collection in embodied manipulation. While UMI enables portable, wrist-mounted data acquisition, its reliance on monocular visual SLAM makes it vulnerable to occlusions, dynamic scenes, and tracking failures, limiting its applicability in real-world environments. UMI-3D addresses these limitations by introducing a lightweight and low-cost LiDAR sensor tightly integrated into the wrist-mounted interface, enabling LiDAR-centric SLAM with accurate metric-scale pose estimation under challenging conditions. We further develop a hardware-synchronized multimodal sensing pipeline and a unified spatiotemporal calibration framework that aligns visual observations with LiDAR point clouds, producing consistent 3D representations of demonstrations. Despite maintaining the original 2D visuomotor policy formulation, UMI-3D significantly improves the quality and reliability of collected data, which directly translates into enhanced policy performance. Extensive real-world experiments demonstrate that UMI-3D not only achieves high success rates on standard manipulation tasks, but also enables learning of tasks that are challenging or infeasible for the original vision-only UMI setup, including large deformable object manipulation and articulated object operation. The system supports an end-to-end pipeline for data acquisition, alignment, training, and deployment, while preserving the portability and accessibility of the original UMI. All hardware and software components are open-sourced to facilitate large-scale data collection and accelerate research in embodied intelligence: hrefhttps://umi-3d.github.iohttps://umi-3d.github.io.