# STTAR: Surgical Tool Tracking using off-the-shelf Augmented Reality Head-Mounted Displays

arXiv: https://arxiv.org/abs/2208.08880 · HTML: https://mlsanity.com/paper/2208.08880/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Aug 2022.

## Built on by (most cited first, 17 of 17)

- [Automatic registration with continuous pose updates for marker-less surgical navigation in spine surgery](https://mlsanity.com/paper/2308.02917/) (2023)
- [MoEdit: On Learning Quantity Perception for Multi-object Image Editing](https://mlsanity.com/paper/2503.10112/) (2025)
- [EVD Surgical Guidance with Retro-Reflective Tool Tracking and Spatial Reconstruction using Head-Mounted Augmented Reality Device](https://mlsanity.com/paper/2306.15490/) (2023)
- [EasyREG: Easy Depth-Based Markerless Registration and Tracking using Augmented Reality Device for Surgical Guidance](https://mlsanity.com/paper/2504.09498/) (2025)
- [SurgiTrack: Fine-Grained Multi-Class Multi-Tool Tracking in Surgical Videos](https://mlsanity.com/paper/2405.20333/) (2024)
- [Navigate Biopsy with Ultrasound under Augmented Reality Device: Towards Higher System Performance](https://mlsanity.com/paper/2402.02414/) (2024)
- [Markerless Augmented Reality Registration for Surgical Guidance: A Multi-Anatomy Clinical Accuracy Study](https://mlsanity.com/paper/2511.02086/) (2025)
- [All-in-One Augmented Reality Guided Head and Neck Tumor Resection](https://mlsanity.com/paper/2603.29495/) (2026)
- [MRUCT: Mixed Reality Assistance for Acupuncture Guided by Ultrasonic Computed Tomography](https://mlsanity.com/paper/2502.08786/) (2025)
- [SurgNavAR: An Augmented Reality Surgical Navigation Framework for Optical See-Through Head Mounted Displays](https://mlsanity.com/paper/2603.29990/) (2026)
- [AR Surgical Navigation with Surface Tracing: Comparing In-Situ Visualization with Tool-Tracking Guidance for Neurosurgical Applications](https://mlsanity.com/paper/2508.10554/) (2025)
- [Impact of Target and Tool Visualization on Depth Perception and Usability in Optical See-Through AR](https://mlsanity.com/paper/2508.18481/) (2025)
- [Multimodal Feedback for Handheld Tool Guidance: Combining Wrist-Based Haptics with Augmented Reality](https://mlsanity.com/paper/2601.12037/) (2026)
- [Extend Your Horizon: A Device-Agnostic Surgical Tool Tracking Framework with Multi-View Optimization for Augmented Reality](https://mlsanity.com/paper/2603.07981/) (2026)
- [Patient4D: Temporally Consistent Patient Body Mesh Recovery from Monocular Operating Room Video](https://mlsanity.com/paper/2603.17178/) (2026)
- [Multi-Camera AR Guidance System for Surgical Instrument Handling and Assembly: Investigating Workload and Efficiency](https://mlsanity.com/paper/2606.04992/) (2026)
- [X-GuideAR: An Augmented Reality Framework to Mitigate Radiation Exposure during Fluoroscopic Guidance](https://mlsanity.com/paper/2607.10873/) (2026)

## Abstract (arXiv)

The use of Augmented Reality (AR) for navigation purposes has shown beneficial in assisting physicians during the performance of surgical procedures. These applications commonly require knowing the pose of surgical tools and patients to provide visual information that surgeons can use during the task performance. Existing medical-grade tracking systems use infrared cameras placed inside the Operating Room (OR) to identify retro-reflective markers attached to objects of interest and compute their pose. Some commercially available AR Head-Mounted Displays (HMDs) use similar cameras for self-localization, hand tracking, and estimating the objects' depth. This work presents a framework that uses the built-in cameras of AR HMDs to enable accurate tracking of retro-reflective markers, such as those used in surgical procedures, without the need to integrate any additional components. This framework is also capable of simultaneously tracking multiple tools. Our results show that the tracking and detection of the markers can be achieved with an accuracy of 0.09 +- 0.06 mm on lateral translation, 0.42 +- 0.32 mm on longitudinal translation, and 0.80 +- 0.39 deg for rotations around the vertical axis. Furthermore, to showcase the relevance of the proposed framework, we evaluate the system's performance in the context of surgical procedures. This use case was designed to replicate the scenarios of k-wire insertions in orthopedic procedures. For evaluation, two surgeons and one biomedical researcher were provided with visual navigation, each performing 21 injections. Results from this use case provide comparable accuracy to those reported in the literature for AR-based navigation procedures.