mlsanityData to 7 Oct 2026

Pixel-Wise Recognition for Holistic Surgical Scene Understanding

arXiv 2401.11174 ↗First posted 20 Jan 2024cs.CV cs.AI cs.LGcited by 48

Not on the map. Why?

Abstract (arXiv)

This paper presents the Holistic and Multi-Granular Surgical Scene Understanding of Prostatectomies (GraSP) dataset, a curated benchmark that models surgical scene understanding as a hierarchy of complementary tasks with varying levels of granularity. Our approach encompasses long-term tasks, such as surgical phase and step recognition, and short-term tasks, including surgical instrument segmentation and atomic visual actions detection. To exploit our proposed benchmark, we introduce the Transformers for Actions, Phases, Steps, and Instrument Segmentation (TAPIS) model, a general architecture that combines a global video feature extractor with localized region proposals from an instrument segmentation model to tackle the multi-granularity of our benchmark. Through extensive experimentation in ours and alternative benchmarks, we demonstrate TAPIS's versatility and state-of-the-art performance across different tasks. This work represents a foundational step forward in Endoscopic Vision, offering a novel framework for future research towards holistic surgical scene understanding.

Key facts. Cited by 48 papers in the map (data through 7 Oct 2026). Builds on 53 of its 107 references that are in the map. First posted on arXiv 20 Jan 2024.

Builds on

53 of 107 references.

References · 53

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  7. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  8. 2020End-to-End Object Detection with Transformerscited by 7,384
  9. 2017Mask R-CNNcited by 7,259
  10. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  11. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  12. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
38 more
  1. 2018SlowFast Networks for Video Recognitioncited by 1,862
  2. 2022DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detectioncited by 1,306
  3. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,148
  4. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  5. 2021Multiscale Vision Transformerscited by 896
  6. 2016Hollywood in Homes: Crowdsourcing Data Collection for Activity Understandingcited by 535
  7. 2017AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actionscited by 449
  8. 2016EndoNet: A Deep Architecture for Recognition Tasks on Laparoscopic Videoscited by 366
  9. 2016YouTube-8M: A Large-Scale Video Classification Benchmarkcited by 324
  10. 2016The THUMOS Challenge on Action Recognition for Videos "in the Wild"cited by 213
  11. 20192017 Robotic Instrument Segmentation Challengecited by 166
  12. 2021Rendezvous: Attention Mechanisms for the Recognition of Surgical Action Triplets in Endoscopic Videoscited by 139
  13. 2017Surgical Data Science: Enabling Next-Generation Surgerycited by 126
  14. 2022Neural Rendering for Stereo 3D Reconstruction of Deformable Tissues in Robotic Surgerycited by 122
  15. 2020Surgical Data Science -- from Concepts toward Clinical Translationcited by 105
  16. 2015Every Moment Counts: Dense Detailed Labeling of Actions in Complex Videoscited by 93
  17. 2021Trans-SVNet: Accurate Phase Recognition from Surgical Videos via Hybrid Embedding Aggregation Transformercited by 93
  18. 2020The AVA-Kinetics Localized Human Actions Video Datasetcited by 82
  19. 2018Automatic Instrument Segmentation in Robot-Assisted Surgery Using Deep Learningcited by 81
  20. 2020ISINet: An Instance-Based Approach for Surgical Instrument Segmentationcited by 67
  21. 2020Recognition of Instrument-Tissue Interactions in Endoscopic Videos via Action Tripletscited by 66
  22. 2019Incorporating Temporal Prior from Motion Flow for Instrument Segmentation in Minimally Invasive Surgery Videocited by 64
  23. 2021OperA: Attention-Regularized Transformers for Surgical Phase Recognitioncited by 60
  24. 2021Comparative Validation of Machine Learning Algorithms for Surgical Workflow and Skill Analysis with the HeiChole Benchmarkcited by 58
  25. 2018DeepPhase: Surgical Phase Recognition in CATARACTS Videoscited by 47
  26. 2020Heidelberg Colorectal Data Set for Surgical Data Science in the Sensor Operating Roomcited by 47
  27. 2022TraSeTR: Track-to-Segment Transformer with Contrastive Query for Instance-level Instrument Segmentation in Robotic Surgerycited by 40
  28. 2016The TUM LapChole dataset for the M2CAI 2016 workflow challengecited by 37
  29. 2021The SARAS Endoscopic Surgeon Action Detection (ESAD) dataset: Challenges and methodscited by 37
  30. 2018Comparative evaluation of instrument segmentation and tracking methods in minimally invasive surgerycited by 36
  31. 2023MATIS: Masked-Attention Transformers for Surgical Instrument Segmentationcited by 36
  32. 2020Automatic Operating Room Surgical Activity Recognition for Robot-Assisted Surgerycited by 34
  33. 2017ToolNet: Holistically-Nested Real-Time Segmentation of Robotic Surgical Toolscited by 30
  34. 2020Learning and Reasoning with the Graph Structure Representation in Robotic Surgerycited by 29
  35. 2022Global-Reasoned Multi-Task Learning Model for Surgical Scene Understandingcited by 29
  36. 2021MIcro-Surgical Anastomose Workflow recognition challenge reportcited by 27
  37. 2022Rendezvous in Time: An Attention-based Temporal Fusion approach for Surgical Triplet Recognitioncited by 27
  38. 2021Exploring Segment-level Semantics for Online Phase Recognition from Surgical Videoscited by 23

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 24; 2026: 22.

48 papers.

Sort:

By area:

  1. 2024Procedure-Aware Surgical Video-language Pretraining with Hierarchical Knowledge Augmentationcited by 38
  2. 2025SurgVLM: A Large Vision-Language Model and Systematic Evaluation Benchmark for Surgical Intelligencecited by 30
  3. 2025LEMON: A Large Endoscopic MONocular Dataset and Foundation Model for Perception in Surgical Settingscited by 19
  4. 2025SurgLaVi: Large-Scale Hierarchical Dataset for Surgical Vision-Language Representation Learningcited by 16
  5. 2025Cosmos-H-Surgical: Learning Surgical Robot Policies from Videos via World Modelingcited by 10
  6. 2024MuST: Multi-Scale Transformers for Surgical Phase Recognitioncited by 8
  7. 2025Surgical Scene Understanding in the Era of Foundation AI Models: A Comprehensive Reviewcited by 8
  8. 2025Comparative validation of surgical phase recognition, instrument keypoint estimation, and instrument instance segmentation in endoscopy: Results of the PhaKIR 2024 challengecited by 6
  9. 2026SUREON: A Benchmark and Vision-Language-Model for Surgical Reasoningcited by 6
  10. 2025ProstaTD: Bridging Surgical Triplet from Classification to Fully Supervised Detectioncited by 4
  11. 2025Spatio-Temporal Representation Decoupling and Enhancement for Federated Instrument Segmentation in Surgical Videoscited by 4
  12. 2025Surgical Action Planning with Large Language Modelscited by 3
36 more
  1. 2025Multimodal Graph Representation Learning for Robust Surgical Workflow Recognition with Adversarial Feature Disentanglementcited by 3
  2. 2025Holistic Surgical Phase Recognition with Hierarchical Input Dependent State Space Modelscited by 3
  3. 2026Where It Moves, It Matters: Referring Surgical Instrument Segmentation via Motioncited by 3
  4. 2026Generalized Recognition of Basic Surgical Actions Enables Skill Assessment and Vision-Language-Model-based Surgical Planningcited by 3
  5. 2026CliPPER: Contextual Video-Language Pretraining on Long-form Intraoperative Surgical Procedures for Event Recognitioncited by 3
  6. 2026MedHorizon: Towards Long-context Medical Video Understanding in the Wildcited by 3
  7. 2025Recognize Any Surgical Object: Unleashing the Power of Weakly-Supervised Datacited by 2
  8. 2025TEMSET-24K: Densely Annotated Dataset for Indexing Multipart Endoscopic Videos using Surgical Timeline Segmentationcited by 2
  9. 2025SurgMLLMBench: A Multimodal Large Language Model Benchmark Dataset for Surgical Scene Understandingcited by 2
  10. 2026Surg-R1: A Hierarchical Reasoning Foundation Model for Scalable and Interpretable Surgical Decision Support with Multi-Center Clinical Validationcited by 2
  11. 2025SAP-Bench: Benchmarking Multimodal Large Language Models in Surgical Action Planningcited by 1
  12. 2025Surgical Video Understanding with Label Interpolationcited by 1
  13. 2025Mitigating Surgical Data Imbalance with Dual-Prediction Video Diffusion Modelcited by 1
  14. 2025Grounding Surgical Action Triplets with Instrument Instance Segmentation: A Dataset and Target-Aware Fusion Approachcited by 1
  15. 2025SurgSLOT: Segment Anything in Surgical Videos via Semantic Long-term Trackingcited by 1
  16. 2026Evaluating Large Vision-language Models for Surgical Tool Detectioncited by 1
  17. 2026SurgSigma: A Spectrum of Large-Scale Multimodal Data and Foundation Models for Surgical Intelligencecited by 1
  18. 2026SurgTEMP: Temporal-Aware Surgical Video Question Answering with Text-guided Visual Memory for Laparoscopic Cholecystectomycited by 1
  19. 2026An Approach to Enriching Surgical Video Datasets for Fine-Grained Spatial-Temporal Understanding of Vision-Language Modelscited by 1
  20. 2026SurgVista: Long-Horizon Surgical World Modeling with Plausible Instrument-Tissue Dynamicscited by 1
  21. 2026SurgAtlas: A Large-Scale Surgical Video-Language Dataset with 2,391 Hours of Open and Minimally Invasive Surgerycited by 1
  22. 2025EndoControlMag: Robust Endoscopic Vascular Motion Magnification with Periodic Reference Resetting and Hierarchical Tissue-aware Dual-Mask Controlcited by 0
  23. 2025SurgPIS: Surgical-instrument-level Instances and Part-level Semantics for Weakly-supervised Part-aware Instance Segmentationcited by 0
  24. 2025Multi-scale Temporal Prediction via Incremental Generation and Multi-agent Collaborationcited by 0
  25. 2025Token Merging via Spatiotemporal Information Mining for Surgical Video Understandingcited by 0
  26. 2025State-Change Learning for Prediction of Future Events in Endoscopic Videoscited by 0
  27. 2026SurgFusion-Net: Diversified Adaptive Multimodal Fusion Network for Surgical Skill Assessmentcited by 0
  28. 2026SurgicalMamba: Dual-Path SSD with State Regramming for Online Surgical Phase Recognitioncited by 0
  29. 2026Temporally Consistent Label Interpolation for Robust Surgical Multi-Task Learning under Challenging Conditionscited by 0
  30. 2026SPIRIT: Spatio-temporal Pairwise Relational Modeling of Instrument-Tissue Interactions for Surgical Action Triplet Recognitioncited by 0
  31. 2026SurgNarrator: A Generative Retrieval Framework for Surgical Video Understandingcited by 0
  32. 2026Toward Mask Annotation-Free Surgical Instrument Segmentation from Endoscopic Images Using Text-Prompted Segment Anything Model 3 (SAM3)cited by 0
  33. 2026TEDi: Temporal Memory-Enhanced and Denoising Transformer for Surgical Instrument Segmentationcited by 0
  34. 2026LD-RSVIS: A Large-Scale and Diverse Benchmark for Referring Surgical Video Instrument Segmentationcited by 0
  35. 2026A Hierarchy-Aware Video-Language Model Evaluation and Hyperbolic Baseline for Surgerycited by 0
  36. 2026A Surgical Foundation Model Reveals Task-Dependent Label Efficiencycited by 0