# HoloLens 2 Sensor Streaming

arXiv: https://arxiv.org/abs/2211.02648 · HTML: https://mlsanity.com/paper/2211.02648/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Nov 2022.

## Built on by (most cited first, 11 of 11)

- [CaptainCook4D: A Dataset for Understanding Errors in Procedural Activities](https://mlsanity.com/paper/2312.14556/) (2023)
- [Instance Tracking in 3D Scenes from Egocentric Videos](https://mlsanity.com/paper/2312.04117/) (2023)
- [A Comparative Neural Radiance Field (NeRF) 3D Analysis of Camera Poses from HoloLens Trajectories and Structure from Motion](https://mlsanity.com/paper/2304.10664/) (2023)
- [HoloPOCUS: Portable Mixed-Reality 3D Ultrasound Tracking, Reconstruction and Overlay](https://mlsanity.com/paper/2308.13823/) (2023)
- [Combining HoloLens with Instant-NeRFs: Advanced Real-Time 3D Mobile Mapping](https://mlsanity.com/paper/2304.14301/) (2023)
- [MultiCam: On-the-fly Multi-Camera Pose Estimation Using Spatiotemporal Overlaps of Known Objects](https://mlsanity.com/paper/2603.22839/) (2026)
- [HOLa: HoloLens Object Labeling](https://mlsanity.com/paper/2412.04945/) (2024)
- [Impact of Target and Tool Visualization on Depth Perception and Usability in Optical See-Through AR](https://mlsanity.com/paper/2508.18481/) (2025)
- [Multimodal Feedback for Handheld Tool Guidance: Combining Wrist-Based Haptics with Augmented Reality](https://mlsanity.com/paper/2601.12037/) (2026)
- [ARport: An Augmented Reality System for Markerless Image-Guided Port Placement in Robotic Surgery](https://mlsanity.com/paper/2602.14153/) (2026)
- [Multi-Camera AR Guidance System for Surgical Instrument Handling and Assembly: Investigating Workload and Efficiency](https://mlsanity.com/paper/2606.04992/) (2026)

## Abstract (arXiv)

We present a HoloLens 2 server application for streaming device data via TCP in real time. The server can stream data from the four grayscale cameras, depth sensor, IMU, front RGB camera, microphone, head tracking, eye tracking, and hand tracking. Each sent data frame has a timestamp and, optionally, the instantaneous pose of the device in 3D space. The server allows downloading device calibration data, such as camera intrinsics, and can be integrated into Unity projects as a plugin, with support for basic upstream capabilities. To achieve real time video streaming at full frame rate, we leverage the video encoding capabilities of the HoloLens 2. Finally, we present a Python library for receiving and decoding the data, which includes utilities that facilitate passing the data to other libraries. The source code, Python demos, and precompiled binaries are available at https://github.com/jdibenes/hl2ss.