# PanORama: Multiview Consistent Panoptic Segmentation in Operating Rooms

arXiv: https://arxiv.org/abs/2603.19920 · HTML: https://mlsanity.com/paper/2603.19920/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 29 references that are in the map.
- First posted on arXiv 20 Mar 2026.

## Builds on (most cited first, 23 of 23)

- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [VGGT: Visual Geometry Grounded Transformer](https://mlsanity.com/paper/2503.11651/) (2025)
- [DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queries](https://mlsanity.com/paper/2110.06922/) (2021)
- [EndoNet: A Deep Architecture for Recognition Tasks on Laparoscopic Videos](https://mlsanity.com/paper/1602.03012/) (2016)
- [Exploring Object-Centric Temporal Modeling for Efficient Multi-View 3D Object Detection](https://mlsanity.com/paper/2303.11926/) (2023)
- [MapAnything: Universal Feed-Forward Metric 3D Reconstruction](https://mlsanity.com/paper/2509.13414/) (2025)
- [Rendezvous: Attention Mechanisms for the Recognition of Surgical Action Triplets in Endoscopic Videos](https://mlsanity.com/paper/2109.03223/) (2021)
- [Surgical Data Science: Enabling Next-Generation Surgery](https://mlsanity.com/paper/1701.06482/) (2017)
- [Multi-Task Recurrent Convolutional Network with Correlation Loss for Surgical Video Analysis](https://mlsanity.com/paper/1907.06099/) (2019)
- [4D-OR: Semantic Scene Graphs for OR Domain Modeling](https://mlsanity.com/paper/2203.11937/) (2022)
- [Advancing Surgical VQA with Scene Graph Knowledge](https://mlsanity.com/paper/2312.10251/) (2023)
- [Ray Denoising: Depth-aware Hard Negative Sampling for Multi-view 3D Object Detection](https://mlsanity.com/paper/2402.03634/) (2024)
- [Automatic Operating Room Surgical Activity Recognition for Robot-Assisted Surgery](https://mlsanity.com/paper/2006.16166/) (2020)
- [GCFAgg: Global and Cross-view Feature Aggregation for Multi-view Clustering](https://mlsanity.com/paper/2305.06799/) (2023)
- [MM-OR: A Large Multimodal Operating Room Dataset for Semantic Understanding of High-Intensity Surgical Environments](https://mlsanity.com/paper/2503.02579/) (2025)
- [PitVQA: Image-grounded Text Embedding LLM for Visual Question Answering in Pituitary Surgery](https://mlsanity.com/paper/2405.13949/) (2024)
- [LABRAD-OR: Lightweight Memory Scene Graphs for Accurate Bimodal Reasoning in Dynamic Operating Rooms](https://mlsanity.com/paper/2303.13293/) (2023)
- [EgoSurgery-Phase: A Dataset of Surgical Phase Recognition from Egocentric Open Surgery Videos](https://mlsanity.com/paper/2405.19644/) (2024)
- [CaRTS: Causality-driven Robot Tool Segmentation from Vision and Kinematics Data](https://mlsanity.com/paper/2203.09475/) (2022)
- [ORacle: Large Vision-Language Models for Knowledge-Guided Holistic OR Domain Modeling](https://mlsanity.com/paper/2404.07031/) (2024)
- [EgoExOR: An Ego-Exo-Centric Operating Room Dataset for Surgical Activity Understanding](https://mlsanity.com/paper/2505.24287/) (2025)
- [Tri-modal Confluence with Temporal Dynamics for Scene Graph Generation in Operating Rooms](https://mlsanity.com/paper/2404.09231/) (2024)
- [Specialized Foundation Models for Intelligent Operating Rooms](https://mlsanity.com/paper/2505.12890/) (2025)

## Abstract (arXiv)

Operating rooms (ORs) are cluttered, dynamic, highly occluded environments, where reliable spatial understanding is essential for situational awareness during complex surgical workflows. Achieving spatial understanding for panoptic segmentation from sparse multiview images poses a fundamental challenge, as limited visibility in a subset of views often leads to mispredictions across cameras. To this end, we introduce PanORama, the first panoptic segmentation for the operating room that is multiview-consistent by design. By modeling cross-view interactions at the feature level inside the backbone in a single forward pass, view consistency emerges directly rather than through post-hoc refinement. We evaluate on the MM-OR and 4D-OR datasets, achieving >70% Panoptic Quality (PQ) performance, and outperforming the previous state of the art. Importantly, PanORama is calibration-free, requiring no camera parameters, and generalizes to unseen camera viewpoints within any multiview configuration at inference time. By substantially enhancing multiview segmentation and, consequently, spatial understanding in the OR, we believe our approach opens new opportunities for surgical perception and assistance. Code will be released upon acceptance.