# EgoExOR: An Ego-Exo-Centric Operating Room Dataset for Surgical Activity Understanding

arXiv: https://arxiv.org/abs/2505.24287 · HTML: https://mlsanity.com/paper/2505.24287/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 47 references that are in the map.
- First posted on arXiv 30 May 2025.

## Builds on (most cited first, 30 of 34)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [Scaling Egocentric Vision: The EPIC-KITCHENS Dataset](https://mlsanity.com/paper/1804.02748/) (2018)
- [Point Transformer V3: Simpler, Faster, Stronger](https://mlsanity.com/paper/2312.10035/) (2023)
- [Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives](https://mlsanity.com/paper/2311.18259/) (2023)
- [EndoNet: A Deep Architecture for Recognition Tasks on Laparoscopic Videos](https://mlsanity.com/paper/1602.03012/) (2016)
- [HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interaction](https://mlsanity.com/paper/2203.01577/) (2022)
- [Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activities](https://mlsanity.com/paper/2203.14712/) (2022)
- [Project Aria: A New Tool for Egocentric Multi-Modal AI Research](https://mlsanity.com/paper/2308.13561/) (2023)
- [Aria Digital Twin: A New Benchmark Dataset for Egocentric 3D Machine Perception](https://mlsanity.com/paper/2306.06362/) (2023)
- [Rendezvous: Attention Mechanisms for the Recognition of Surgical Action Triplets in Endoscopic Videos](https://mlsanity.com/paper/2109.03223/) (2021)
- [Surgical Data Science: Enabling Next-Generation Surgery](https://mlsanity.com/paper/1701.06482/) (2017)
- [HD-EPIC: A Highly-Detailed Egocentric Video Dataset](https://mlsanity.com/paper/2502.04144/) (2025)
- [Recognition of Instrument-Tissue Interactions in Endoscopic Videos via Action Triplets](https://mlsanity.com/paper/2007.05405/) (2020)
- [OperA: Attention-Regularized Transformers for Surgical Phase Recognition](https://mlsanity.com/paper/2103.03873/) (2021)
- [Tool Detection and Operative Skill Assessment in Surgical Videos Using Region-Based Convolutional Neural Networks](https://mlsanity.com/paper/1802.08774/) (2018)
- [MVOR: A Multi-view RGB-D Operating Room Dataset for 2D and 3D Human Pose Estimation](https://mlsanity.com/paper/1808.08180/) (2018)
- [4D-OR: Semantic Scene Graphs for OR Domain Modeling](https://mlsanity.com/paper/2203.11937/) (2022)
- [EPIC Fields: Marrying 3D Geometry and Video Understanding](https://mlsanity.com/paper/2306.08731/) (2023)
- [Advancing Surgical VQA with Scene Graph Knowledge](https://mlsanity.com/paper/2312.10251/) (2023)
- [Automatic Operating Room Surgical Activity Recognition for Robot-Assisted Surgery](https://mlsanity.com/paper/2006.16166/) (2020)
- [Learning and Reasoning with the Graph Structure Representation in Robotic Surgery](https://mlsanity.com/paper/2007.03357/) (2020)
- [MM-OR: A Large Multimodal Operating Room Dataset for Semantic Understanding of High-Intensity Surgical Environments](https://mlsanity.com/paper/2503.02579/) (2025)
- [Dynamic Scene Graph Representation for Surgical Video](https://mlsanity.com/paper/2309.14538/) (2023)
- [EgoSurgery-Phase: A Dataset of Surgical Phase Recognition from Egocentric Open Surgery Videos](https://mlsanity.com/paper/2405.19644/) (2024)

## Built on by (most cited first, 11 of 11)

- [EgoExo-Con: Exploring View-Invariant Video Temporal Understanding](https://mlsanity.com/paper/2510.26113/) (2025)
- [Egosurg: Arbitrary view synthesis for egocentric replay of operating room workflows from ambient cameras](https://mlsanity.com/paper/2510.04802/) (2025)
- [RocSync: Millisecond-Accurate Temporal Synchronization for Heterogeneous Camera Systems](https://mlsanity.com/paper/2511.14948/) (2025)
- [Mistake Attribution: Fine-Grained Mistake Understanding in Egocentric Videos](https://mlsanity.com/paper/2511.20525/) (2025)
- [ProcObject-10K: Benchmarking Object-Centric Procedural Understanding in Instructional Videos](https://mlsanity.com/paper/2512.03479/) (2025)
- [Benchmarking Egocentric Clinical Intent Understanding Capability for Medical Multimodal Large Language Models](https://mlsanity.com/paper/2601.06750/) (2026)
- [Sound Source Localization for Spatial Mapping of Surgical Actions in Dynamic Scenes](https://mlsanity.com/paper/2510.24332/) (2025)
- [PanORama: Multiview Consistent Panoptic Segmentation in Operating Rooms](https://mlsanity.com/paper/2603.19920/) (2026)
- [Where are they looking in the operating room?](https://mlsanity.com/paper/2604.20574/) (2026)
- [OR-Action: Multi-Role Video Understanding with Fine-Grained Actions](https://mlsanity.com/paper/2606.13332/) (2026)
- [HEIR: Learning Human-Entity Interactions with Functional Roles](https://mlsanity.com/paper/2609.35955/) (2026)

## Abstract (arXiv)

Operating rooms (ORs) demand precise coordination among surgeons, nurses, and equipment in a fast-paced, occlusion-heavy environment, necessitating advanced perception models to enhance safety and efficiency. Existing datasets either provide partial egocentric views or sparse exocentric multi-view context, but do not explore the comprehensive combination of both. We introduce EgoExOR, the first OR dataset and accompanying benchmark to fuse first-person and third-person perspectives. Spanning 94 minutes (84,553 frames at 15 FPS) of two emulated spine procedures, Ultrasound-Guided Needle Insertion and Minimally Invasive Spine Surgery, EgoExOR integrates egocentric data (RGB, gaze, hand tracking, audio) from wearable glasses, exocentric RGB and depth from RGB-D cameras, and ultrasound imagery. Its detailed scene graph annotations, covering 36 entities and 22 relations (568,235 triplets), enable robust modeling of clinical interactions, supporting tasks like action recognition and human-centric perception. We evaluate the surgical scene graph generation performance of two adapted state-of-the-art models and offer a new baseline that explicitly leverages EgoExOR's multimodal and multi-perspective signals. This new dataset and benchmark set a new foundation for OR perception, offering a rich, multimodal resource for next-generation clinical perception.