# Virtual Reality via Object Pose Estimation and Active Learning: Realizing Telepresence Robots with Aerial Manipulation Capabilities

arXiv: https://arxiv.org/abs/2210.09678 · HTML: https://mlsanity.com/paper/2210.09678/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 103 references that are in the map.
- First posted on arXiv 18 Oct 2022.

## Builds on (most cited first, 30 of 30)

- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [A Survey of Uncertainty in Deep Neural Networks](https://mlsanity.com/paper/2107.03342/) (2021)
- [Deep Closest Point: Learning Representations for Point Cloud Registration](https://mlsanity.com/paper/1905.03304/) (2019)
- [Learning High-Speed Flight in the Wild](https://mlsanity.com/paper/2110.05113/) (2021)
- [A Review and Comparative Study on Probabilistic Object Detection in Autonomous Driving](https://mlsanity.com/paper/2011.10671/) (2020)
- [Active Learning for Deep Object Detection via Probabilistic Modeling](https://mlsanity.com/paper/2103.16130/) (2021)
- [BayesOD: A Bayesian Approach for Uncertainty Estimation in Deep Object Detectors](https://mlsanity.com/paper/1903.03838/) (2019)
- [Active Learning for Deep Detection Neural Networks](https://mlsanity.com/paper/1911.09168/) (2019)
- [NeBula: Quest for Robotic Autonomy in Challenging Environments; TEAM CoSTAR at the DARPA Subterranean Challenge](https://mlsanity.com/paper/2103.11470/) (2021)
- [Estimating Model Uncertainty of Neural Networks in Sparse Information Form](https://mlsanity.com/paper/2006.11631/) (2020)
- [Heterogeneous Ground and Air Platforms, Homogeneous Sensing: Team CSIRO Data61's Approach to the DARPA Subterranean Challenge](https://mlsanity.com/paper/2104.09053/) (2021)
- [CERBERUS: Autonomous Legged and Aerial Robotic Exploration in the Tunnel and Urban Circuits of the DARPA Subterranean Challenge](https://mlsanity.com/paper/2201.07067/) (2022)
- [Trust Your Robots! Predictive Uncertainty Estimation of Neural Networks with Sparse Gaussian Processes](https://mlsanity.com/paper/2109.09690/) (2021)
- [Present and Future of SLAM in Extreme Underground Environments](https://mlsanity.com/paper/2208.01787/) (2022)
- [Analysis of Robust Functions for Registration Algorithms](https://mlsanity.com/paper/1810.01474/) (2018)
- [Bayesian Active Learning for Sim-to-Real Robotic Perception](https://mlsanity.com/paper/2109.11547/) (2021)
- [GraspLook: a VR-based Telemanipulation System with R-CNN-driven Augmentation of Virtual Environment](https://mlsanity.com/paper/2110.12518/) (2021)
- [Bayesian Optimization Meets Laplace Approximation for Robotic Introspection](https://mlsanity.com/paper/2010.16141/) (2020)
- [System for multi-robotic exploration of underground environments CTU-CRAS-NORLAB in the DARPA Subterranean Challenge](https://mlsanity.com/paper/2110.05911/) (2021)
- [Visual-Inertial Telepresence for Aerial Manipulation](https://mlsanity.com/paper/2003.11509/) (2020)
- [Development of SAM: cable-Suspended Aerial Manipulator](https://mlsanity.com/paper/1903.02426/) (2019)
- [Dynamic Grasping with a "Soft" Drone: From Theory to Practice](https://mlsanity.com/paper/2103.06465/) (2021)
- [Long-Duration Fully Autonomous Operation of Rotorcraft Unmanned Aerial Systems for Remote-Sensing Data Acquisition](https://mlsanity.com/paper/1908.06381/) (2019)
- [What the HoloLens Maps Is Your Workspace: Fast Mapping and Set-up of Robot Cells via Head Mounted Displays and Augmented Reality](https://mlsanity.com/paper/2005.12651/) (2020)
- [Automated acquisition of structured, semantic models of manipulation activities from human VR demonstration](https://mlsanity.com/paper/2011.13689/) (2020)
- [AeroVR: Virtual Reality-based Teleoperation with Tactile Feedback for Aerial Manipulation](https://mlsanity.com/paper/1910.11604/) (2019)
- [An Augmented Reality Interaction Interface for Autonomous Drone](https://mlsanity.com/paper/2008.02234/) (2020)

## Built on by (most cited first, 1 of 1)

- [SPIRIT: Perceptive Shared Autonomy for Robust Robotic Manipulation under Deep Learning Uncertainty](https://mlsanity.com/paper/2603.05111/) (2026)

## Abstract (arXiv)

This article presents a novel telepresence system for advancing aerial manipulation in dynamic and unstructured environments. The proposed system not only features a haptic device, but also a virtual reality (VR) interface that provides real-time 3D displays of the robot's workspace as well as a haptic guidance to its remotely located operator. To realize this, multiple sensors namely a LiDAR, cameras and IMUs are utilized. For processing of the acquired sensory data, pose estimation pipelines are devised for industrial objects of both known and unknown geometries. We further propose an active learning pipeline in order to increase the sample efficiency of a pipeline component that relies on Deep Neural Networks (DNNs) based object detection. All these algorithms jointly address various challenges encountered during the execution of perception tasks in industrial scenarios. In the experiments, exhaustive ablation studies are provided to validate the proposed pipelines. Methodologically, these results commonly suggest how an awareness of the algorithms' own failures and uncertainty (`introspection') can be used tackle the encountered problems. Moreover, outdoor experiments are conducted to evaluate the effectiveness of the overall system in enhancing aerial manipulation capabilities. In particular, with flight campaigns over days and nights, from spring to winter, and with different users and locations, we demonstrate over 70 robust executions of pick-and-place, force application and peg-in-hole tasks with the DLR cable-Suspended Aerial Manipulator (SAM). As a result, we show the viability of the proposed system in future industrial applications.