mlsanityData to 7 Oct 2026

Towards Holistic Surgical Scene Understanding

arXiv 2212.04582 ↗First posted 8 Dec 2022cs.CV cs.AIcited by 33

Not on the map. Why?

Abstract (arXiv)

Most benchmarks for studying surgical interventions focus on a specific challenge instead of leveraging the intrinsic complementarity among different tasks. In this work, we present a new experimental framework towards holistic surgical scene understanding. First, we introduce the Phase, Step, Instrument, and Atomic Visual Action recognition (PSI-AVA) Dataset. PSI-AVA includes annotations for both long-term (Phase and Step recognition) and short-term reasoning (Instrument detection and novel Atomic Action recognition) in robot-assisted radical prostatectomy videos. Second, we present Transformers for Action, Phase, Instrument, and steps Recognition (TAPIR) as a strong baseline for surgical scene understanding. TAPIR leverages our dataset's multi-level annotations as it benefits from the learned representation on the instrument detection task to improve its classification capacity. Our experimental results in both PSI-AVA and other publicly available databases demonstrate the adequacy of our framework to spur future research on holistic surgical scene understanding.

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). Builds on 27 of its 34 references that are in the map. First posted on arXiv 8 Dec 2022.

Builds on

27 of 34 references.

References · 27

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  4. 2020End-to-End Object Detection with Transformerscited by 7,384
  5. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  6. 2017The Kinetics Human Action Video Datasetcited by 1,894
  7. 2018SlowFast Networks for Video Recognitioncited by 1,862
  8. 2021Video Swin Transformercited by 1,090
  9. 2021Multiscale Vision Transformerscited by 896
  10. 2017AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actionscited by 449
  11. 2016EndoNet: A Deep Architecture for Recognition Tasks on Laparoscopic Videoscited by 366
  12. 20192017 Robotic Instrument Segmentation Challengecited by 166
15 more
  1. 2021Rendezvous: Attention Mechanisms for the Recognition of Surgical Action Triplets in Endoscopic Videoscited by 139
  2. 2017Surgical Data Science: Enabling Next-Generation Surgerycited by 126
  3. 20202018 Robotic Scene Segmentation Challengecited by 123
  4. 2021Trans-SVNet: Accurate Phase Recognition from Surgical Videos via Hybrid Embedding Aggregation Transformercited by 93
  5. 2020ISINet: An Instance-Based Approach for Surgical Instrument Segmentationcited by 67
  6. 2020Recognition of Instrument-Tissue Interactions in Endoscopic Videos via Action Tripletscited by 66
  7. 2021OperA: Attention-Regularized Transformers for Surgical Phase Recognitioncited by 60
  8. 2020Heidelberg Colorectal Data Set for Surgical Data Science in the Sensor Operating Roomcited by 47
  9. 2022TraSeTR: Track-to-Segment Transformer with Contrastive Query for Instance-level Instrument Segmentation in Robotic Surgerycited by 40
  10. 2016The TUM LapChole dataset for the M2CAI 2016 workflow challengecited by 37
  11. 2021The SARAS Endoscopic Surgeon Action Detection (ESAD) dataset: Challenges and methodscited by 37
  12. 2020Automatic Operating Room Surgical Activity Recognition for Robot-Assisted Surgerycited by 34
  13. 2021MIcro-Surgical Anastomose Workflow recognition challenge reportcited by 27
  14. 2019CaDIS: Cataract Dataset for Image Segmentationcited by 17
  15. 2021A real-time spatiotemporal AI model analyzes skill in open surgical videoscited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 7; 2024: 10; 2025: 12; 2026: 3.

33 papers.

Sort:

By area:

  1. 2023SurgicalGPT: End-to-End Language-Vision GPT for Visual Question Answering in Surgerycited by 48
  2. 2023MATIS: Masked-Attention Transformers for Surgical Instrument Segmentationcited by 36
  3. 2025Scaling up self-supervised learning for improved surgical foundation modelscited by 25
  4. 2025SurgLaVi: Large-Scale Hierarchical Dataset for Surgical Vision-Language Representation Learningcited by 16
  5. 2024General-purpose foundation models for increased autonomy in robot-assisted surgerycited by 15
  6. 2024Surgical-LLaVA: Toward Surgical Scenario Understanding via Large Language and Vision Modelscited by 13
  7. 2025EndoBench: A Comprehensive Evaluation of Multi-Modal Large Language Models for Endoscopy Analysiscited by 13
  8. 2025From Screens to Scenes: A Survey of Embodied AI in Healthcarecited by 10
  9. 2025Large-scale Self-supervised Video Foundation Model for Intelligent Surgerycited by 10
  10. 2023Whether and When does Endoscopy Domain Pretraining Make Sense?cited by 8
  11. 2023Pelphix: Surgical Phase Recognition from X-ray Images in Percutaneous Pelvic Fixationcited by 8
  12. 2024Instrument-tissue Interaction Detection Framework for Surgical Video Understandingcited by 8
21 more
  1. 2024MuST: Multi-Scale Transformers for Surgical Phase Recognitioncited by 8
  2. 2024Exploring the Effect of Dataset Diversity in Self-Supervised Learning for Surgical Computer Visioncited by 7
  3. 2024Memory-Augmented Multimodal LLMs for Surgical VQA via Self-Contained Inquirycited by 6
  4. 2023SF-TMN: SlowFast Temporal Modeling Network for Surgical Phase Recognitioncited by 5
  5. 2023RIDE: Self-Supervised Learning of Rotation-Equivariant Keypoint Detection and Invariant Description for Endoscopycited by 5
  6. 2024Multitask Learning in Minimally Invasive Surgical Vision: A Reviewcited by 5
  7. 2024Neural Finite-State Machines for Surgical Phase Recognitioncited by 5
  8. 2025ProstaTD: Bridging Surgical Triplet from Classification to Fully Supervised Detectioncited by 4
  9. 2026SurgMotion: A Video-Native Foundation Model for Universal Understanding of Surgical Videoscited by 4
  10. 2022MURPHY: Relations Matter in Surgical Workflow Analysiscited by 3
  11. 2026From Articulated Kinematics to Routed Visual Control for Action-Conditioned Surgical Video Generationcited by 3
  12. 2024Strategies to Improve Real-World Applicability of Laparoscopic Anatomy Segmentation Modelscited by 2
  13. 2025TEMSET-24K: Densely Annotated Dataset for Indexing Multipart Endoscopic Videos using Surgical Timeline Segmentationcited by 2
  14. 2025Hierarchical Context Transformer for Multi-level Semantic Scene Understandingcited by 2
  15. 2025MoSFormer: Augmenting Temporal Context with Memory of Surgery for Surgical Phase Recognitioncited by 2
  16. 2025Towards Holistic Surgical Scene Graphcited by 2
  17. 2024SURGIVID: Annotation-Efficient Surgical Video Object Discoverycited by 1
  18. 2025Surgical Video Understanding with Label Interpolationcited by 1
  19. 2023SurGNN: Explainable visual scene understanding and assessment of surgical skill using graph neural networkscited by 0
  20. 2025State-Change Learning for Prediction of Future Events in Endoscopic Videoscited by 0
  21. 2026Strategy-Supervised Autonomous Laparoscopic Camera Control via Event-Driven Graph Miningcited by 0