mlsanityData to 7 Oct 2026

OphNet: A Large-Scale Video Benchmark for Ophthalmic Surgical Workflow Understanding

arXiv 2406.07471 ↗First posted 11 Jun 2024cs.CVcited by 31

Not on the map. Why?

Abstract (arXiv)

Surgical scene perception via videos is critical for advancing robotic surgery, telesurgery, and AI-assisted surgery, particularly in ophthalmology. However, the scarcity of diverse and richly annotated video datasets has hindered the development of intelligent systems for surgical workflow analysis. Existing datasets face challenges such as small scale, lack of diversity in surgery and phase categories, and absence of time-localized annotations. These limitations impede action understanding and model generalization validation in complex and diverse real-world surgical scenarios. To address this gap, we introduce OphNet, a large-scale, expert-annotated video benchmark for ophthalmic surgical workflow understanding. OphNet features: 1) A diverse collection of 2,278 surgical videos spanning 66 types of cataract, glaucoma, and corneal surgeries, with detailed annotations for 102 unique surgical phases and 150 fine-grained operations. 2) Sequential and hierarchical annotations for each surgery, phase, and operation, enabling comprehensive understanding and improved interpretability. 3) Time-localized annotations, facilitating temporal localization and prediction tasks within surgical workflows. With approximately 285 hours of surgical videos, OphNet is about 20 times larger than the largest existing surgical workflow analysis benchmark. Code and dataset are available at: https://minghu0830.github.io/OphNet-benchmark/.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Builds on 58 of its 99 references that are in the map. First posted on arXiv 11 Jun 2024.

Builds on

58 of 99 references.

References · 58

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  3. 2017The Kinetics Human Action Video Datasetcited by 1,894
  4. 2018SlowFast Networks for Video Recognitioncited by 1,862
  5. 2020X3D: Expanding Architectures for Efficient Video Recognitioncited by 635
  6. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 616
  7. 2016Hollywood in Homes: Crowdsourcing Data Collection for Activity Understandingcited by 535
  8. 2021MViTv2: Improved Multiscale Vision Transformers for Classification and Detectioncited by 516
  9. 2017AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actionscited by 449
  10. 2016EndoNet: A Deep Architecture for Recognition Tasks on Laparoscopic Videoscited by 366
  11. 2019BMN: Boundary-Matching Network for Temporal Action Proposal Generationcited by 327
  12. 2022Expanding Language-Image Pretrained Models for General Video Recognitioncited by 320
38 more
  1. 2022ActionFormer: Localizing Moments of Actions with Transformerscited by 308
  2. 2019Video Classification with Channel-Separated Convolutional Networkscited by 290
  3. 2019G-TAD: Sub-Graph Localization for Temporal Action Detectioncited by 248
  4. 2018Human Action Recognition and Prediction: A Surveycited by 226
  5. 2020The EPIC-KITCHENS Dataset: Collection, Challenges and Baselinescited by 221
  6. 2019Graph Convolutional Networks for Temporal Action Localizationcited by 208
  7. 2022Fine-tuned CLIP Models are Efficient Video Learnerscited by 178
  8. 2023TriDet: Temporal Action Detection with Relative Boundary Modelingcited by 141
  9. 2020TeCNO: Surgical Phase Recognition with Multi-Stage Temporal Convolutional Networkscited by 139
  10. 2021Rendezvous: Attention Mechanisms for the Recognition of Surgical Action Triplets in Endoscopic Videoscited by 139
  11. 2019What Would You Expect? Anticipating Egocentric Actions with Rolling-Unrolling LSTMs and Modality Attentioncited by 104
  12. 2020Video Self-Stitching Graph Network for Temporal Action Localizationcited by 104
  13. 2022AutoLaparo: A New Dataset of Integrated Multi-tasks for Image-guided Surgical Automation in Laparoscopic Hysterectomycited by 85
  14. 2022Video Polyp Segmentation: A Deep Learning Perspectivecited by 81
  15. 2020TSP: Temporally-Sensitive Pretraining of Video Encoders for Localization Taskscited by 77
  16. 2020Spatiotemporal Relationship Reasoning for Pedestrian Intent Predictioncited by 71
  17. 2022Revisiting Classifier: Transferring Vision-Language Models for Video Recognitioncited by 70
  18. 2023EndoSurf: Neural Surface Reconstruction of Deformable Tissues with Stereo Endoscope Videoscited by 68
  19. 2023Learning Multi-modal Representations by Watching Hundreds of Surgical Video Lecturescited by 68
  20. 2020Recognition of Instrument-Tissue Interactions in Endoscopic Videos via Action Tripletscited by 66
  21. 2021Enriching Local and Global Contexts for Temporal Action Localizationcited by 64
  22. 2021OperA: Attention-Regularized Transformers for Surgical Phase Recognitioncited by 60
  23. 2021Comparative Validation of Machine Learning Algorithms for Surgical Workflow and Skill Analysis with the HeiChole Benchmarkcited by 58
  24. 2018Anticipating Traffic Accidents with Adaptive Loss and Large-scale Incident DBcited by 51
  25. 2022Bidirectional Cross-Modal Knowledge Exploration for Video Recognition with Pre-trained Vision-Language Modelscited by 48
  26. 2020Heidelberg Colorectal Data Set for Surgical Data Science in the Sensor Operating Roomcited by 47
  27. 2024HecVL: Hierarchical Video-Language Pretraining for Zero-shot Surgical Phase Recognitioncited by 41
  28. 2021Self-Supervised Learning for Semi-Supervised Temporal Action Proposalcited by 39
  29. 2016The TUM LapChole dataset for the M2CAI 2016 workflow challengecited by 37
  30. 2017Exploiting the potential of unlabeled endoscopic video data with self-supervised learningcited by 33
  31. 2019RAUNet: Residual Attention U-Net for Semantic Segmentation of Cataract Surgical Instrumentscited by 27
  32. 2018RSDNet: Learning to Predict Remaining Surgery Duration from Laparoscopic Videos Without Manual Annotationscited by 26
  33. 2018Less is More: Surgical Phase Recognition with Less Annotations through Self-Supervised Pre-training of CNN-LSTM Networkscited by 24
  34. 2018Learning from a tiny dataset of manual annotations: a teacher/student approach for surgical phase recognitioncited by 21
  35. 2023Prompt-Guided Zero-Shot Anomaly Action Recognition using Pretrained Deep Skeleton Featurescited by 21
  36. 2022Contrastive Transformer-based Multiple Instance Learning for Weakly Supervised Polyp Frame Detectioncited by 20
  37. 2023NurViD: A Large Expert-Level Video Database for Nursing Procedure Activity Understandingcited by 20
  38. 2022Semantic-SuPer: A Semantic-aware Surgical Perception Framework for Endoscopic Tissue Identification, Reconstruction, and Trackingcited by 19

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 15; 2026: 8.

31 papers.

Sort:

By area:

  1. 2024MMed-RAG: Versatile Multimodal RAG System for Medical Vision Language Modelscited by 84
  2. 2024RULE: Reliable Multimodal RAG for Factuality in Medical Vision Language Modelscited by 58
  3. 2025SurgVLM: A Large Vision-Language Model and Systematic Evaluation Benchmark for Surgical Intelligencecited by 30
  4. 2024OphCLIP: Hierarchical Retrieval-Augmented Learning for Ophthalmic Surgical Video-Language Pretrainingcited by 29
  5. 2025MMRC: A Large-Scale Benchmark for Understanding Multimodal Large Language Model in Real-World Conversationcited by 28
  6. 2025MM-OR: A Large Multimodal Operating Room Dataset for Semantic Understanding of High-Intensity Surgical Environmentscited by 28
  7. 2024MMedPO: Aligning Medical Vision-Language Models with Clinical-Aware Multimodal Preference Optimizationcited by 21
  8. 2024Large Language Model Benchmarks in Medical Taskscited by 16
  9. 2024TP-DRSeg: Improving Diabetic Retinopathy Lesion Segmentation with Explicit Text-Prompts Assisted SAMcited by 13
  10. 2024Frontiers in Intelligent Colonoscopycited by 13
  11. 2025UniMedVL: Unifying Medical Multimodal Understanding and Generation through Observation-Knowledge-Analysiscited by 12
  12. 2026Advances in Global Solvers for 3D Visioncited by 12
19 more
  1. 2025Ophora: A Large-Scale Data-Driven Text-Guided Ophthalmic Surgical Video Generation Modelcited by 8
  2. 2024LMOD: A Large Multimodal Ophthalmology Dataset and Benchmark for Large Vision-Language Modelscited by 7
  3. 2025CLINIC: Evaluating Multilingual Trustworthiness in Language Models for Healthcarecited by 7
  4. 2025SG2VID: Scene Graphs Enable Fine-Grained Control for Video Synthesiscited by 4
  5. 2026SurgMotion: A Video-Native Foundation Model for Universal Understanding of Surgical Videoscited by 4
  6. 2025EyePCR: A Comprehensive Benchmark for Fine-Grained Perception, Knowledge Comprehension and Clinical Reasoning in Ophthalmic Surgerycited by 3
  7. 2026MedHorizon: Towards Long-context Medical Video Understanding in the Wildcited by 3
  8. 2025Recognize Any Surgical Object: Unleashing the Power of Weakly-Supervised Datacited by 2
  9. 2025Towards Dynamic 3D Reconstruction of Hand-Instrument Interaction in Ophthalmic Surgerycited by 2
  10. 2026Towards Next-Generation Healthcare: A Survey of Medical Embodied AI for Perception, Decision-Making, and Actioncited by 2
  11. 2025RDTF: Resource-efficient Dual-mask Training Framework for Multi-frame Animated Sticker Generationcited by 1
  12. 2025LMOD+: A Comprehensive Multimodal Dataset and Benchmark for Developing and Evaluating Multimodal Large Language Models in Ophthalmologycited by 1
  13. 2026SurgTEMP: Temporal-Aware Surgical Video Question Answering with Text-guided Visual Memory for Laparoscopic Cholecystectomycited by 1
  14. 2025CoStoDet-DDPM: Collaborative Training of Stochastic and Deterministic Models Improves Surgical Workflow Anticipation and Recognitioncited by 0
  15. 2025WetCat: Enabling Automated Skill Assessment in Wet-Lab Cataract Surgery Videoscited by 0
  16. 2025Meta-SurDiff: Classification Diffusion Model Optimized by Meta Learning is Reliable for Online Surgical Phase Recognitioncited by 0
  17. 2026SurgCoT: Advancing Spatiotemporal Reasoning in Surgical Videos through a Chain-of-Thought Benchmarkcited by 0
  18. 2026Multimodal Shared Latent Representation of Narration, Microscope and iOCT Images for Phase Recognition in Vitreoretinal Surgerycited by 0
  19. 2026OphBiWSSD: Scaling Temporal Action Localization in Ophthalmic Surgeries with Bidirectional Weight-tied State Space Dualitycited by 0