# DeepMix: Mobility-aware, Lightweight, and Hybrid 3D Object Detection for Headsets

arXiv: https://arxiv.org/abs/2201.08812 · HTML: https://mlsanity.com/paper/2201.08812/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 67 references that are in the map.
- First posted on arXiv 15 Jan 2022.

## Builds on (most cited first, 26 of 26)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [PointPillars: Fast Encoders for Object Detection from Point Clouds](https://mlsanity.com/paper/1812.05784/) (2018)
- [YOLOv4: Optimal Speed and Accuracy of Object Detection](https://mlsanity.com/paper/2004.10934/) (2020)
- [VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection](https://mlsanity.com/paper/1711.06396/) (2017)
- [PointRCNN: 3D Object Proposal Generation and Detection from Point Cloud](https://mlsanity.com/paper/1812.04244/) (2018)
- [PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detection](https://mlsanity.com/paper/1912.13192/) (2019)
- [Multi-View 3D Object Detection Network for Autonomous Driving](https://mlsanity.com/paper/1611.07759/) (2016)
- [Deep Hough Voting for 3D Object Detection in Point Clouds](https://mlsanity.com/paper/1904.09664/) (2019)
- [Frustum PointNets for 3D Object Detection from RGB-D Data](https://mlsanity.com/paper/1711.08488/) (2017)
- [Joint 3D Proposal Generation and Object Detection from View Aggregation](https://mlsanity.com/paper/1712.02294/) (2017)
- [Learning Depth-Guided Convolutions for Monocular 3D Object Detection](https://mlsanity.com/paper/1912.04799/) (2019)
- [Accurate Monocular Object Detection via Color-Embedded 3D Reconstruction for Autonomous Driving](https://mlsanity.com/paper/1903.11444/) (2019)
- [ClearGrasp: 3D Shape Estimation of Transparent Objects for Manipulation](https://mlsanity.com/paper/1910.02550/) (2019)
- [ImVoteNet: Boosting 3D Object Detection in Point Clouds with Image Votes](https://mlsanity.com/paper/2001.10692/) (2020)
- [Deep Sliding Shapes for Amodal 3D Object Detection in RGB-D Images](https://mlsanity.com/paper/1511.02300/) (2015)
- [3DIoUMatch: Leveraging IoU Prediction for Semi-Supervised 3D Object Detection](https://mlsanity.com/paper/2012.04355/) (2020)
- [MLCVNet: Multi-Level Context VoteNet for 3D Object Detection](https://mlsanity.com/paper/2004.05679/) (2020)
- [Vote3Deep: Fast Object Detection in 3D Point Clouds Using Efficient Convolutional Neural Networks](https://mlsanity.com/paper/1609.06666/) (2016)
- [Deep Fitting Degree Scoring Network for Monocular 3D Object Detection](https://mlsanity.com/paper/1904.12681/) (2019)
- [3D Object Proposals using Stereo Imagery for Accurate Object Class Detection](https://mlsanity.com/paper/1608.07711/) (2016)
- [Transferable Semi-supervised 3D Object Detection from RGB-D Data](https://mlsanity.com/paper/1904.10300/) (2019)
- [Mesorasi: Architecture Support for Point Cloud Analytics via Delayed-Aggregation](https://mlsanity.com/paper/2008.06967/) (2020)
- [A Survey of Calibration Methods for Optical See-Through Head-Mounted Displays](https://mlsanity.com/paper/1709.04299/) (2017)

## Built on by (most cited first, 10 of 10)

- [Artificial Intelligence of Things: A Survey](https://mlsanity.com/paper/2410.19998/) (2024)
- [Real-Time Onboard Object Detection for Augmented Reality: Enhancing Head-Mounted Display with YOLOv8](https://mlsanity.com/paper/2306.03537/) (2023)
- [Panopticus: Omnidirectional 3D Object Detection on Resource-constrained Edge Devices](https://mlsanity.com/paper/2410.01270/) (2024)
- [Training an Open-Vocabulary Monocular 3D Object Detection Model without 3D Data](https://mlsanity.com/paper/2411.15657/) (2024)
- [PointSplit: Towards On-device 3D Object Detection with Heterogeneous Low-power Accelerators](https://mlsanity.com/paper/2504.03654/) (2025)
- [Augmented Efficiency: Reducing Memory Footprint and Accelerating Inference for 3D Semantic Segmentation through Hybrid Vision](https://mlsanity.com/paper/2407.16102/) (2024)
- [t-READi: Transformer-Powered Robust and Efficient Multimodal Inference for Autonomous Driving](https://mlsanity.com/paper/2410.09747/) (2024)
- [KHAIT: K-9 Handler Artificial Intelligence Teaming for Collaborative Sensemaking](https://mlsanity.com/paper/2503.15524/) (2025)
- [Deep Learning Inference on Heterogeneous Mobile Processors: Potentials and Pitfalls](https://mlsanity.com/paper/2405.01851/) (2024)
- [EvenNICER-SLAM: Event-based Neural Implicit Encoding SLAM](https://mlsanity.com/paper/2410.03812/) (2024)

## Abstract (arXiv)

Mobile headsets should be capable of understanding 3D physical environments to offer a truly immersive experience for augmented/mixed reality (AR/MR). However, their small form-factor and limited computation resources make it extremely challenging to execute in real-time 3D vision algorithms, which are known to be more compute-intensive than their 2D counterparts. In this paper, we propose DeepMix, a mobility-aware, lightweight, and hybrid 3D object detection framework for improving the user experience of AR/MR on mobile headsets. Motivated by our analysis and evaluation of state-of-the-art 3D object detection models, DeepMix intelligently combines edge-assisted 2D object detection and novel, on-device 3D bounding box estimations that leverage depth data captured by headsets. This leads to low end-to-end latency and significantly boosts detection accuracy in mobile scenarios. A unique feature of DeepMix is that it fully exploits the mobility of headsets to fine-tune detection results and boost detection accuracy. To the best of our knowledge, DeepMix is the first 3D object detection that achieves 30 FPS (an end-to-end latency much lower than the 100 ms stringent requirement of interactive AR/MR). We implement a prototype of DeepMix on Microsoft HoloLens and evaluate its performance via both extensive controlled experiments and a user study with 30+ participants. DeepMix not only improves detection accuracy by 9.1--37.3% but also reduces end-to-end latency by 2.68--9.15x, compared to the baseline that uses existing 3D object detection models.