# Real-Time Onboard Object Detection for Augmented Reality: Enhancing Head-Mounted Display with YOLOv8

arXiv: https://arxiv.org/abs/2306.03537 · HTML: https://mlsanity.com/paper/2306.03537/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 46 references that are in the map.
- First posted on arXiv 6 Jun 2023.

## Builds on (most cited first, 9 of 9)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Object Detection in 20 Years: A Survey](https://mlsanity.com/paper/1905.05055/) (2019)
- [A Survey of Modern Deep Learning based Object Detection Models](https://mlsanity.com/paper/2104.11892/) (2021)
- [A Full Dive into Realizing the Edge-enabled Metaverse: Visions, Enabling Technologies,and Challenges](https://mlsanity.com/paper/2203.05471/) (2022)
- [When Internet of Things meets Metaverse: Convergence of Physical and Cyber Worlds](https://mlsanity.com/paper/2208.13501/) (2022)
- [DeepMix: Mobility-aware, Lightweight, and Hybrid 3D Object Detection for Headsets](https://mlsanity.com/paper/2201.08812/) (2022)
- [MAGES 4.0: Accelerating the world's transition to VR training and democratizing the authoring of the medical metaverse](https://mlsanity.com/paper/2209.08819/) (2022)
- [A Framework of Transfer Learning in Object Detection for Embedded Systems](https://mlsanity.com/paper/1811.04863/) (2018)

## Built on by (most cited first, 3 of 3)

- [GO-N3RDet: Geometry Optimized NeRF-enhanced 3D Object Detector](https://mlsanity.com/paper/2503.15211/) (2025)
- [Towards certifiable AI in aviation: landscape, challenges, and opportunities](https://mlsanity.com/paper/2409.08666/) (2024)
- [PILAR: Personalizing Augmented Reality Interactions with LLM-based Human-Centric and Trustworthy Explanations for Daily Use Cases](https://mlsanity.com/paper/2512.17172/) (2025)

## Abstract (arXiv)

This paper introduces a software architecture for real-time object detection using machine learning (ML) in an augmented reality (AR) environment. Our approach uses the recent state-of-the-art YOLOv8 network that runs onboard on the Microsoft HoloLens 2 head-mounted display (HMD). The primary motivation behind this research is to enable the application of advanced ML models for enhanced perception and situational awareness with a wearable, hands-free AR platform. We show the image processing pipeline for the YOLOv8 model and the techniques used to make it real-time on the resource-limited edge computing platform of the headset. The experimental results demonstrate that our solution achieves real-time processing without needing offloading tasks to the cloud or any other external servers while retaining satisfactory accuracy regarding the usual mAP metric and measured qualitative performance