# Sound Source Localization for Spatial Mapping of Surgical Actions in Dynamic Scenes

arXiv: https://arxiv.org/abs/2510.24332 · HTML: https://mlsanity.com/paper/2510.24332/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 29 references that are in the map.
- First posted on arXiv 28 Oct 2025.

## Builds on (most cited first, 15 of 15)

- [AST: Audio Spectrogram Transformer](https://mlsanity.com/paper/2104.01778/) (2021)
- [Surgical Data Science: Enabling Next-Generation Surgery](https://mlsanity.com/paper/1701.06482/) (2017)
- [Learning Multi-modal Representations by Watching Hundreds of Surgical Video Lectures](https://mlsanity.com/paper/2307.15220/) (2023)
- [Recognition of Instrument-Tissue Interactions in Endoscopic Videos via Action Triplets](https://mlsanity.com/paper/2007.05405/) (2020)
- [OperA: Attention-Regularized Transformers for Surgical Phase Recognition](https://mlsanity.com/paper/2103.03873/) (2021)
- [4D-OR: Semantic Scene Graphs for OR Domain Modeling](https://mlsanity.com/paper/2203.11937/) (2022)
- [MM-OR: A Large Multimodal Operating Room Dataset for Semantic Understanding of High-Intensity Surgical Environments](https://mlsanity.com/paper/2503.02579/) (2025)
- [LABRAD-OR: Lightweight Memory Scene Graphs for Accurate Bimodal Reasoning in Dynamic Operating Rooms](https://mlsanity.com/paper/2303.13293/) (2023)
- [ASI-Seg: Audio-Driven Surgical Instrument Segmentation with Surgeon Intention Understanding](https://mlsanity.com/paper/2407.19435/) (2024)
- [EgoExOR: An Ego-Exo-Centric Operating Room Dataset for Surgical Activity Understanding](https://mlsanity.com/paper/2505.24287/) (2025)
- [Creating a Digital Twin of Spinal Surgery: A Proof of Concept](https://mlsanity.com/paper/2403.16736/) (2024)
- [From Decision to Action in Surgical Autonomy: Multi-Modal Large Language Models for Robot-Assisted Blood Suction](https://mlsanity.com/paper/2408.07806/) (2024)
- [ST(OR)2: Spatio-Temporal Object Level Reasoning for Activity Recognition in the Operating Room](https://mlsanity.com/paper/2312.12250/) (2023)
- [Conditional Generative Data Augmentation for Clinical Audio Datasets](https://mlsanity.com/paper/2203.11570/) (2022)
- [Automatic breach detection during spine pedicle drilling based on vibroacoustic sensing](https://mlsanity.com/paper/2303.15114/) (2023)

## Abstract (arXiv)

Purpose: Surgical scene understanding is key to advancing computer-aided and intelligent surgical systems. Current approaches predominantly rely on visual data or end-to-end learning, which limits fine-grained contextual modeling. This work aims to enhance surgical scene representations by integrating 3D acoustic information, enabling temporally and spatially aware multimodal understanding of surgical environments. Methods: We propose a novel framework for generating 4D audio-visual representations of surgical scenes by projecting acoustic localization information from a phased microphone array onto dynamic point clouds from an RGB-D camera. A transformer-based acoustic event detection module identifies relevant temporal segments containing tool-tissue interactions which are spatially localized in the audio-visual scene representation. The system was experimentally evaluated in a realistic operating room setup during simulated surgical procedures performed by experts. Results: The proposed method successfully localizes surgical acoustic events in 3D space and associates them with visual scene elements. Experimental evaluation demonstrates accurate spatial sound localization and robust fusion of multimodal data, providing a comprehensive, dynamic representation of surgical activity. Conclusion: This work introduces the first approach for spatial sound localization in dynamic surgical scenes, marking a significant advancement toward multimodal surgical scene representations. By integrating acoustic and visual data, the proposed framework enables richer contextual understanding and provides a foundation for future intelligent and autonomous surgical systems.