# HoloLens 2 Research Mode as a Tool for Computer Vision Research

arXiv: https://arxiv.org/abs/2008.11239 · HTML: https://mlsanity.com/paper/2008.11239/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Aug 2020.

## Built on by (most cited first, 30 of 34)

- [EgoBody: Human Body Shape and Motion of Interacting People from Head-Mounted Devices](https://mlsanity.com/paper/2112.07642/) (2021)
- [HOT3D: Hand and Object Tracking in 3D from Egocentric Multi-View Videos](https://mlsanity.com/paper/2411.19167/) (2024)
- [LaMAR: Benchmarking Localization and Mapping for Augmented Reality](https://mlsanity.com/paper/2210.10770/) (2022)
- [GIMO: Gaze-Informed Human Motion Prediction in Context](https://mlsanity.com/paper/2204.09443/) (2022)
- [FLAG: Flow-based 3D Avatar Generation from Sparse Observations](https://mlsanity.com/paper/2203.05789/) (2022)
- [IndustReal: A Dataset for Procedure Step Recognition Handling Execution Errors in Egocentric Videos in an Industrial-Like Setting](https://mlsanity.com/paper/2310.17323/) (2023)
- [HMD-NeMo: Online 3D Avatar Motion Generation From Sparse Observations](https://mlsanity.com/paper/2308.11261/) (2023)
- [Instance Tracking in 3D Scenes from Egocentric Videos](https://mlsanity.com/paper/2312.04117/) (2023)
- [STEPs: Self-Supervised Key Step Extraction and Localization from Unlabeled Procedural Videos](https://mlsanity.com/paper/2301.00794/) (2023)
- [Context-Aware Sequence Alignment using 4D Skeletal Augmentation](https://mlsanity.com/paper/2204.12223/) (2022)
- [The Monado SLAM Dataset for Egocentric Visual-Inertial Tracking](https://mlsanity.com/paper/2508.00088/) (2025)
- [InfraredTags: Embedding Invisible AR Markers and Barcodes Using Low-Cost, Infrared-Based 3D Printing and Imaging Tools](https://mlsanity.com/paper/2202.06165/) (2022)
- [Lazy Visual Localization via Motion Averaging](https://mlsanity.com/paper/2307.09981/) (2023)
- [SIGMA: An Open-Source Interactive System for Mixed-Reality Task Assistance Research](https://mlsanity.com/paper/2405.13035/) (2024)
- [The HoloLens in Medicine: A systematic Review and Taxonomy](https://mlsanity.com/paper/2209.03245/) (2022)
- [EVD Surgical Guidance with Retro-Reflective Tool Tracking and Spatial Reconstruction using Head-Mounted Augmented Reality Device](https://mlsanity.com/paper/2306.15490/) (2023)
- [Human-Centric Resource Allocation for the Metaverse With Multiaccess Edge Computing](https://mlsanity.com/paper/2312.15313/) (2023)
- [HoloPOCUS: Portable Mixed-Reality 3D Ultrasound Tracking, Reconstruction and Overlay](https://mlsanity.com/paper/2308.13823/) (2023)
- [Navigate Biopsy with Ultrasound under Augmented Reality Device: Towards Higher System Performance](https://mlsanity.com/paper/2402.02414/) (2024)
- [AI-Powered Augmented Reality for Satellite Assembly, Integration and Test](https://mlsanity.com/paper/2409.18101/) (2024)
- [MR.NAVI: Mixed-Reality Navigation Assistant for the Visually Impaired](https://mlsanity.com/paper/2506.05369/) (2025)
- [DeltaDorsal: Enhancing Hand Pose Estimation with Dorsal Features in Egocentric Views](https://mlsanity.com/paper/2601.15516/) (2026)
- [MultiCam: On-the-fly Multi-Camera Pose Estimation Using Spatiotemporal Overlaps of Known Objects](https://mlsanity.com/paper/2603.22839/) (2026)
- [SurgNavAR: An Augmented Reality Surgical Navigation Framework for Optical See-Through Head Mounted Displays](https://mlsanity.com/paper/2603.29990/) (2026)
- [VOCA: Visual Odometry with Codec Awareness](https://mlsanity.com/paper/2607.00189/) (2026)
- [SHREC 2021: Track on Skeleton-based Hand Gesture Recognition in the Wild](https://mlsanity.com/paper/2106.10980/) (2021)
- [PWR-Align: Leveraging Part-Whole Relationships for Part-wise Rigid Point Cloud Registration in Mixed Reality Applications](https://mlsanity.com/paper/2306.06717/) (2023)
- [ARPOV: Expanding Visualization of Object Detection in AR with Panoramic Mosaic Stitching](https://mlsanity.com/paper/2410.01055/) (2024)
- [EvenNICER-SLAM: Event-based Neural Implicit Encoding SLAM](https://mlsanity.com/paper/2410.03812/) (2024)
- [Lost in Tracking Translation: A Comprehensive Analysis of Visual SLAM in Human-Centered XR and IoT Ecosystems](https://mlsanity.com/paper/2411.07146/) (2024)

## Abstract (arXiv)

Mixed reality headsets, such as the Microsoft HoloLens 2, are powerful sensing devices with integrated compute capabilities, which makes it an ideal platform for computer vision research. In this technical report, we present HoloLens 2 Research Mode, an API and a set of tools enabling access to the raw sensor streams. We provide an overview of the API and explain how it can be used to build mixed reality applications based on processing sensor data. We also show how to combine the Research Mode sensor data with the built-in eye and hand tracking capabilities provided by HoloLens 2. By releasing the Research Mode API and a set of open-source tools, we aim to foster further research in the fields of computer vision as well as robotics and encourage contributions from the research community.