# Multi-Camera AR Guidance System for Surgical Instrument Handling and Assembly: Investigating Workload and Efficiency

arXiv: https://arxiv.org/abs/2606.04992 · HTML: https://mlsanity.com/paper/2606.04992/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 29 references that are in the map.
- First posted on arXiv 3 Jun 2026.

## Builds on (most cited first, 12 of 12)

- [YOLOX: Exceeding YOLO Series in 2021](https://mlsanity.com/paper/2107.08430/) (2021)
- [CosyPose: Consistent multi-view multi-object 6D pose estimation](https://mlsanity.com/paper/2008.08465/) (2020)
- [Wide-Depth-Range 6D Object Pose Estimation in Space](https://mlsanity.com/paper/2104.00337/) (2021)
- [RTMO: Towards High-Performance One-Stage Real-Time Multi-Person Pose Estimation](https://mlsanity.com/paper/2312.07526/) (2023)
- [STTAR: Surgical Tool Tracking using off-the-shelf Augmented Reality Head-Mounted Displays](https://mlsanity.com/paper/2208.08880/) (2022)
- [HoloLens 2 Sensor Streaming](https://mlsanity.com/paper/2211.02648/) (2022)
- [Automatic registration with continuous pose updates for marker-less surgical navigation in spine surgery](https://mlsanity.com/paper/2308.02917/) (2023)
- [GBOT: Graph-Based 3D Object Tracking for Augmented Reality-Assisted Assembly Guidance](https://mlsanity.com/paper/2402.07677/) (2024)
- [ASDF: Assembly State Detection Utilizing Late Fusion by Integrating 6D Pose Estimation](https://mlsanity.com/paper/2403.16400/) (2024)
- [NAVIUS: Navigated Augmented Reality Visualization for Ureteroscopic Surgery](https://mlsanity.com/paper/2503.17511/) (2025)
- [Augmented Reality-based Guidance with Deformable Registration in Head and Neck Tumor Resection](https://mlsanity.com/paper/2503.08802/) (2025)
- [MultiCam: On-the-fly Multi-Camera Pose Estimation Using Spatiotemporal Overlaps of Known Objects](https://mlsanity.com/paper/2603.22839/) (2026)

## Abstract (arXiv)

The handling and assembly of instruments during surgery imposes high cognitive demands on scrub nurses, particularly when instruments are unfamiliar. We present a supporting guidance system for surgical instrumentation that combines multi-camera 6D pose estimation with augmented reality in-situ visualization on a head-mounted display without the requirement for additional markers. Pose estimation and consecutive camera calibration are achieved through known objects. The 6D pose estimation network is trained purely on synthetic data, aiming for better generalizability and real-world applicability. The AR guidance displays tooltip localization cues and step-wise assembly animations. Via gaze-based selection and a foot pedal, users can switch between assembly steps in intraoperative use. In a technical evaluation, our approach outperforms state-of-art 6D pose estimation. A user study with 29 scrub nurses was conducted in a surgical simulation of knee arthroplasty, comparing the system against a paper manual. AR guidance significantly reduced the perceived workload compared. Objectively, AR guidance reduced task completion time by 21.3% (4.76 minutes). Specifically, scrub nurses less experienced with the instrument set benefited when using the system. Error frequencies were comparable between conditions. Qualitative feedback highlighted improved process clarity, reduced information overload, and perceived independence. To summarize, our marker-free multi-camera AR guidance approach for surgical instruments can, subjectively and objectively, improve intraoperative instrumentation performance, particularly for untrained scrub nurses.