# Operating Room Workflow Analysis via Reasoning Segmentation over Digital Twins

arXiv: https://arxiv.org/abs/2503.21054 · HTML: https://mlsanity.com/paper/2503.21054/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in Clinical and biomedical applications, one of 17 directions in LLM reasoning.
- Builds on 15 of its 19 references that are in the map.
- First posted on arXiv 26 Mar 2025.

## Placed in

- LLM reasoning / [Clinical and biomedical applications](https://mlsanity.com/lr/clinical-and-biomedical-applications/)

## Builds on (most cited first, 15 of 15)

- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [Depth Anything V2](https://mlsanity.com/paper/2406.09414/) (2024)
- [LISA: Reasoning Segmentation via Large Language Model](https://mlsanity.com/paper/2308.00692/) (2023)
- [V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs](https://mlsanity.com/paper/2312.14135/) (2023)
- [Scaling Open-Vocabulary Object Detection](https://mlsanity.com/paper/2306.09683/) (2023)
- [GSVA: Generalized Segmentation via Multimodal Large Language Models](https://mlsanity.com/paper/2312.10103/) (2023)
- [Deep Learning vs. Traditional Computer Vision](https://mlsanity.com/paper/1910.13796/) (2019)
- [LISA++: An Improved Baseline for Reasoning Segmentation with Large Language Model](https://mlsanity.com/paper/2312.17240/) (2023)
- [LLM-Seg: Bridging Image Segmentation and Large Language Model Reasoning](https://mlsanity.com/paper/2404.08767/) (2024)
- [MVOR: A Multi-view RGB-D Operating Room Dataset for 2D and 3D Human Pose Estimation](https://mlsanity.com/paper/1808.08180/) (2018)
- [Understanding the Performance and Estimating the Cost of LLM Fine-Tuning](https://mlsanity.com/paper/2408.04693/) (2024)
- [Fine Tuning LLM for Enterprise: Practical Guidelines and Recommendations](https://mlsanity.com/paper/2404.10779/) (2024)
- [Reasoning3D -- Grounding and Reasoning in 3D: Fine-Grained Zero-Shot Open-Vocabulary 3D Reasoning Part Segmentation via Large Vision-Language Models](https://mlsanity.com/paper/2405.19326/) (2024)

## Built on by (most cited first, 9 of 9)

- [Reasoning Segmentation for Images and Videos: A Survey](https://mlsanity.com/paper/2505.18816/) (2025)
- [Position: Foundation Models Need Digital Twin Representations](https://mlsanity.com/paper/2505.03798/) (2025)
- [RVTBench: A Benchmark for Visual Reasoning Tasks](https://mlsanity.com/paper/2505.11838/) (2025)
- [Reasoning Text-to-Video Retrieval via Digital Twin Video Representations and Large Language Models](https://mlsanity.com/paper/2511.12371/) (2025)
- [Text-Driven Reasoning Video Editing via Reinforcement Learning on Digital Twin Representations](https://mlsanity.com/paper/2511.14100/) (2025)
- [Temporally-Constrained Video Reasoning Segmentation and Automated Benchmark Construction](https://mlsanity.com/paper/2507.16718/) (2025)
- [Constructing and Interpreting Digital Twin Representations for Visual Reasoning via Reinforcement Learning](https://mlsanity.com/paper/2511.12365/) (2025)
- [Fast Reasoning Segmentation for Images and Videos](https://mlsanity.com/paper/2511.12368/) (2025)
- [When Text Misleads: Inconsistent-Aware Reasoning for Audio-Grounded Dialogue](https://mlsanity.com/paper/2608.27176/) (2026)

## Abstract (arXiv)

Analyzing operating room (OR) workflows to derive quantitative insights into OR efficiency is important for hospitals to maximize patient care and financial sustainability. Prior work on OR-level workflow analysis has relied on end-to-end deep neural networks. While these approaches work well in constrained settings, they are limited to the conditions specified at development time and do not offer the flexibility necessary to accommodate the OR workflow analysis needs of various OR scenarios (e.g., large academic center vs. rural provider) without data collection, annotation, and retraining. Reasoning segmentation (RS) based on foundation models offers this flexibility by enabling automated analysis of OR workflows from OR video feeds given only an implicit text query related to the objects of interest. Due to the reliance on large language model (LLM) fine-tuning, current RS approaches struggle with reasoning about semantic/spatial relationships and show limited generalization to OR video due to variations in visual characteristics and domain-specific terminology. To address these limitations, we first propose a novel digital twin (DT) representation that preserves both semantic and spatial relationships between the various OR components. Then, building on this foundation, we propose ORDiRS (Operating Room Digital twin representation for Reasoning Segmentation), an LLM-tuning-free RS framework that reformulates RS into a "reason-retrieval-synthesize" paradigm. Finally, we present ORDiRS-Agent, an LLM-based agent that decomposes OR workflow analysis queries into manageable RS sub-queries and generates responses by combining detailed textual explanations with supporting visual evidence from RS. Experimental results on both an in-house and a public OR dataset demonstrate that our ORDiRS achieves a cIoU improvement of 6.12%-9.74% compared to the existing state-of-the-arts.