# Extend Your Horizon: A Device-Agnostic Surgical Tool Tracking Framework with Multi-View Optimization for Augmented Reality

arXiv: https://arxiv.org/abs/2603.07981 · HTML: https://mlsanity.com/paper/2603.07981/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 8 of its 35 references that are in the map.
- First posted on arXiv 9 Mar 2026.

## Builds on (most cited first, 8 of 8)

- [Simple Online and Realtime Tracking](https://mlsanity.com/paper/1602.00763/) (2016)
- [Distractor-aware Siamese Networks for Visual Object Tracking](https://mlsanity.com/paper/1808.06048/) (2018)
- [3D Dynamic Scene Graphs: Actionable Spatial Perception with Places, Objects, and Humans](https://mlsanity.com/paper/2002.06289/) (2020)
- [Online Multi-Target Tracking Using Recurrent Neural Networks](https://mlsanity.com/paper/1604.03635/) (2016)
- [Recurrent Autoregressive Networks for Online Multi-Object Tracking](https://mlsanity.com/paper/1711.02741/) (2017)
- [Retargetable AR: Context-aware Augmented Reality in Indoor Scenes based on 3D Scene Graph](https://mlsanity.com/paper/2008.07817/) (2020)
- [STTAR: Surgical Tool Tracking using off-the-shelf Augmented Reality Head-Mounted Displays](https://mlsanity.com/paper/2208.08880/) (2022)
- [Tackling Occlusion in Siamese Tracking with Structured Dropouts](https://mlsanity.com/paper/2006.16571/) (2020)

## Abstract (arXiv)

Surgical navigation provides real-time guidance by estimating the pose of patient anatomy and surgical instruments to visualize relevant intraoperative information. In conventional systems, instruments are typically tracked using fiducial markers and stationary optical tracking systems (OTS). Augmented reality (AR) has further enabled intuitive visualization and motivated tracking using sensors embedded in head-mounted displays (HMDs). However, most existing approaches rely on a clear line of sight, which is difficult to maintain in dynamic operating room environments due to frequent occlusions caused by equipment, surgical tools, and personnel. This work introduces a framework for tracking surgical instruments under occlusion by fusing multiple sensing modalities within a dynamic scene graph representation. The proposed approach integrates tracking systems with different accuracy levels and motion characteristics while estimating tracking reliability in real time. Experimental results demonstrate improved robustness and enhanced consistency of AR visualization in the presence of occlusions.