# Combining HoloLens with Instant-NeRFs: Advanced Real-Time 3D Mobile Mapping

arXiv: https://arxiv.org/abs/2304.14301 · HTML: https://mlsanity.com/paper/2304.14301/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 41 references that are in the map.
- First posted on arXiv 27 Apr 2023.

## Builds on (most cited first, 22 of 22)

- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields](https://mlsanity.com/paper/2111.12077/) (2021)
- [NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction](https://mlsanity.com/paper/2106.10689/) (2021)
- [Plenoxels: Radiance Fields without Neural Networks](https://mlsanity.com/paper/2112.05131/) (2021)
- [TensoRF: Tensorial Radiance Fields](https://mlsanity.com/paper/2203.09517/) (2022)
- [Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations](https://mlsanity.com/paper/1906.01618/) (2019)
- [Depth-supervised NeRF: Fewer Views and Faster Training for Free](https://mlsanity.com/paper/2107.02791/) (2021)
- [BARF: Bundle-Adjusting Neural Radiance Fields](https://mlsanity.com/paper/2104.06405/) (2021)
- [DeepVoxels: Learning Persistent 3D Feature Embeddings](https://mlsanity.com/paper/1812.01024/) (2018)
- [Decomposing NeRF for Editing via Feature Field Distillation](https://mlsanity.com/paper/2205.15585/) (2022)
- [Extracting Triangular 3D Models, Materials, and Lighting From Images](https://mlsanity.com/paper/2111.12503/) (2021)
- [Urban Radiance Fields](https://mlsanity.com/paper/2111.14643/) (2021)
- [Self-Calibrating Neural Radiance Fields](https://mlsanity.com/paper/2108.13826/) (2021)
- [Deblur-NeRF: Neural Radiance Fields from Blurry Images](https://mlsanity.com/paper/2111.14292/) (2021)
- [NeRF: Neural Radiance Field in 3D Vision: A Comprehensive Review (Updated Post-Gaussian Splatting)](https://mlsanity.com/paper/2210.00379/) (2022)
- [RobustNeRF: Ignoring Distractors with Robust Losses](https://mlsanity.com/paper/2302.00833/) (2023)
- [T"oRF: Time-of-Flight Radiance Fields for Dynamic Scene View Synthesis](https://mlsanity.com/paper/2109.15271/) (2021)
- [Neural Volume Rendering: NeRF And Beyond](https://mlsanity.com/paper/2101.05204/) (2020)
- [HoloLens 2 Sensor Streaming](https://mlsanity.com/paper/2211.02648/) (2022)
- [A Comparative Neural Radiance Field (NeRF) 3D Analysis of Camera Poses from HoloLens Trajectories and Structure from Motion](https://mlsanity.com/paper/2304.10664/) (2023)
- [Corrosion Detection for Industrial Objects: From Multi-Sensor System to 5D Feature Space](https://mlsanity.com/paper/2205.07075/) (2022)

## Built on by (most cited first, 1 of 1)

- [An Outlook into the Future of Egocentric Vision](https://mlsanity.com/paper/2308.07123/) (2023)

## Abstract (arXiv)

This work represents a large step into modern ways of fast 3D reconstruction based on RGB camera images. Utilizing a Microsoft HoloLens 2 as a multisensor platform that includes an RGB camera and an inertial measurement unit for SLAM-based camera-pose determination, we train a Neural Radiance Field (NeRF) as a neural scene representation in real-time with the acquired data from the HoloLens. The HoloLens is connected via Wifi to a high-performance PC that is responsible for the training and 3D reconstruction. After the data stream ends, the training is stopped and the 3D reconstruction is initiated, which extracts a point cloud of the scene. With our specialized inference algorithm, five million scene points can be extracted within 1 second. In addition, the point cloud also includes radiometry per point. Our method of 3D reconstruction outperforms grid point sampling with NeRFs by multiple orders of magnitude and can be regarded as a complete real-time 3D reconstruction method in a mobile mapping setup.