mlsanityData to 14 Oct 2026

Surgical Scene Understanding in the Era of Foundation AI Models: A Comprehensive Review

arXiv 2502.14886 ↗First posted 16 Feb 2025cs.CVcited by 8

Not on the map. Why?

Abstract (arXiv)

Recent advancements in machine learning (ML) and deep learning (DL), particularly through the introduction of Foundation Models (FMs), have significantly enhanced surgical scene understanding within minimally invasive surgery (MIS). This paper surveys the integration of state-of-the-art ML and DL technologies, including Convolutional Neural Networks (CNNs), Vision Transformers (ViTs), and Foundation Models like the Segment Anything Model (SAM), into surgical workflows. These technologies improve segmentation accuracy, instrument tracking, and phase recognition in surgical scene understanding. The paper explores the challenges these technologies face, such as data variability and computational demands, and discusses ethical considerations and integration hurdles in clinical settings. Highlighting the roles of FMs, we bridge the technological capabilities with clinical needs and outline future research directions to enhance the adaptability, efficiency, and ethical alignment of AI applications in surgery. Our findings suggest that substantial progress has been made; however, more focused efforts are required to achieve seamless integration of these technologies into clinical workflows, ensuring they complement surgical practice by enhancing precision, reducing risks, and optimizing patient outcomes.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 166 of its 347 references that are in the map. First posted on arXiv 16 Feb 2025.

Builds on

166 of 347 references.

References · 166

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  7. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  8. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  9. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  10. 2023Segment Anythingcited by 8,234
  11. 2020End-to-End Object Detection with Transformerscited by 7,414
  12. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
38 more
  1. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,859
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  3. 2024SAM 2: Segment Anything in Images and Videoscited by 2,863
  4. 2016V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentationcited by 2,292
  5. 2017Rethinking Atrous Convolution for Semantic Image Segmentationcited by 2,161
  6. 2020FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidencecited by 2,085
  7. 2018Attention U-Net: Learning Where to Look for the Pancreascited by 1,435
  8. 2017A Survey on Deep Learning in Medical Image Analysiscited by 1,186
  9. 2017CyCADA: Cycle-Consistent Adversarial Domain Adaptationcited by 1,082
  10. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,013
  11. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 798
  12. 2019Multimodal Transformer for Unaligned Multimodal Language Sequencescited by 645
  13. 2017Unsupervised Anomaly Detection with Generative Adversarial Networks to Guide Marker Discoverycited by 566
  14. 2016Temporal Convolutional Networks for Action Segmentation and Detectioncited by 508
  15. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 429
  16. 2016Deeper Depth Prediction with Fully Convolutional Residual Networkscited by 403
  17. 2016EndoNet: A Deep Architecture for Recognition Tasks on Laparoscopic Videoscited by 367
  18. 2021Big Self-Supervised Models Advance Medical Image Classificationcited by 320
  19. 2022Medical Image Segmentation Review: The success of U-Netcited by 191
  20. 2020U-Net and its variants for medical image segmentation: theory and applicationscited by 169
  21. 2020Medical Image Segmentation Using Deep Learning: A Surveycited by 164
  22. 2023Segment Anything Model (SAM) for Digital Pathology: Assess Zero-shot Segmentation on Whole Slide Imagingcited by 161
  23. 2020Anomaly Detection in Video via Self-Supervised and Multi-Task Learningcited by 148
  24. 2020TeCNO: Surgical Phase Recognition with Multi-Stage Temporal Convolutional Networkscited by 141
  25. 2023On the Challenges and Perspectives of Foundation Models for Medical Image Analysiscited by 135
  26. 2020Deep Learning for Medical Anomaly Detection -- A Surveycited by 127
  27. 2023Large language models in healthcare and medical domain: A reviewcited by 127
  28. 2017Surgical Data Science: Enabling Next-Generation Surgerycited by 126
  29. 20202018 Robotic Scene Segmentation Challengecited by 124
  30. 2019Synergistic Image and Feature Adaptation: Towards Cross-Modality Domain Adaptation for Medical Image Segmentationcited by 121
  31. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 110
  32. 2019Towards Explainable Artificial Intelligencecited by 106
  33. 2023SurgicalSAM: Efficient Class Promptable Surgical Instrument Segmentationcited by 104
  34. 2024Segment Anything Model for Medical Image Segmentation: Current Applications and Future Directionscited by 97
  35. 2019Fast Interactive Object Annotation with Curve-GCNcited by 95
  36. 2022AutoLaparo: A New Dataset of Integrated Multi-tasks for Image-guided Surgical Automation in Laparoscopic Hysterectomycited by 85
  37. 2023MA-SAM: Modality-agnostic SAM Adaptation for 3D Medical Image Segmentationcited by 82
  38. 2018Automatic Instrument Segmentation in Robot-Assisted Surgery Using Deep Learningcited by 81

50 most cited of 166 All 166 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 5.

8 papers.

  1. 2025Surgery-R1: Advancing Surgical-VQLA with Reasoning Multimodal Large Language Model via Reinforcement Learningcited by 5
  2. 2025SurgMLLMBench: A Multimodal Large Language Model Benchmark Dataset for Surgical Scene Understandingcited by 3
  3. 2026Evaluating Large Vision-language Models for Surgical Tool Detectioncited by 2
  4. 2025Token Merging via Spatiotemporal Information Mining for Surgical Video Understandingcited by 0
  5. 2026DeGenseGS: Geometrically and Semantically Decoupled Surgical Scene Understanding in 4D Gaussian Splattingcited by 0
  6. 2026STSG-VQA: Evidence-Grounded Temporal Question Answering from Surgical Spatio-Temporal Scene Graphscited by 0
  7. 2026A Comprehensive Review of Generative Physical Artificial Intelligencecited by 0
  8. 2026DADP: Dynamic Activity-Dependent Pruning, A Reverse Hebbian-Inspired Structural Pruning Methodcited by 0