mlsanityData to 7 Oct 2026

CoPESD: A Multi-Level Surgical Motion Dataset for Training Large Vision-Language Models to Co-Pilot Endoscopic Submucosal Dissection

arXiv 2410.07540 ↗First posted 10 Oct 2024cs.CVcited by 15

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

submucosal dissection (ESD) enables rapid resection of large lesions, minimizing recurrence rates and improving long-term overall survival. Despite these advantages, ESD is technically challenging and carries high risks of complications, necessitating skilled surgeons and precise instruments. Recent advancements in Large Visual-Language Models (LVLMs) offer promising decision support and predictive planning capabilities for robotic systems, which can augment the accuracy of ESD and reduce procedural risks. However, existing datasets for multi-level fine-grained ESD surgical motion understanding are scarce and lack detailed annotations. In this paper, we design a hierarchical decomposition of ESD motion granularity and introduce a multi-level surgical motion dataset (CoPESD) for training LVLMs as the robotic Co-Pilot of Endoscopic Submucosal Dissection. CoPESD includes 17,679 images with 32,699 bounding boxes and 88,395 multi-level motions, from over 35 hours of ESD videos for both robot-assisted and conventional surgeries. CoPESD enables granular analysis of ESD motions, focusing on the complex task of submucosal dissection. Extensive experiments on the LVLMs demonstrate the effectiveness of CoPESD in training LVLMs to predict following surgical robotic motions. As the first multimodal ESD motion dataset, CoPESD supports advanced research in ESD instruction-following and surgical automation. The dataset is available at hrefhttps://github.com/gkw0010/CoPESDhttps://github.com/gkw0010/CoPESD.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 41 of its 74 references that are in the map. First posted on arXiv 10 Oct 2024.

Builds on

41 of 74 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024GP-VLS: A general-purpose vision language model for surgery27 Jul
  2. 2024Surgical-LVLM: Learning to Adapt Large Vision-Language Model for Grounded Visual Question Answering in Robotic Surgery22 Mar
  3. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day1 Jun

Other references · 38

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  6. 2022A ConvNet for the 2020scited by 3,545
  7. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  8. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  9. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  10. 2018Datasheets for Datasetscited by 1,371
  11. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  12. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
26 more
  1. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  2. 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 564
  3. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 517
  4. 2023Vision-Language Foundation Models as Effective Robot Imitatorscited by 344
  5. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 283
  6. 2020Language Conditioned Imitation Learning over Unstructured Datacited by 208
  7. 2023Language-Driven Representation Learning for Roboticscited by 163
  8. 2020Language-Conditioned Imitation Learning for Robot Manipulation Taskscited by 136
  9. 2024SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Modelscited by 130
  10. 2021Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotationcited by 117
  11. 2017Mapping Instructions and Visual Observations to Actions with Reinforcement Learningcited by 80
  12. 2017Interactively Picking Real-World Objects with Unconstrained Spoken Language Instructionscited by 61
  13. 2021StructFormer: Learning Spatial Structure for Language-Guided Semantic Rearrangement of Novel Objectscited by 57
  14. 2018Interactive Visual Grounding of Referring Expressions for Human-Robot Interactioncited by 55
  15. 2018Mapping Navigation Instructions to Continuous Control Actions with Position-Visitation Predictioncited by 35
  16. 2021INVIGORATE: Interactive Visual Grounding and Grasping in Cluttercited by 32
  17. 2020Robot Object Retrieval with Contextual Natural Language Queriescited by 26
  18. 2018Interactive Grounded Language Acquisition and Generalization in a 2D Worldcited by 21
  19. 2021Composing Pick-and-Place Tasks By Grounding Languagecited by 20
  20. 2020Spatial Reasoning from Natural Language Instructions for Robot Manipulationcited by 16
  21. 2023LLCaps: Learning to Illuminate Low-Light Capsule Endoscopy with Curved Wavelet Attention and Reverse Diffusioncited by 15
  22. 2019A New Tensioning Method using Deep Reinforcement Learning for Surgical Pattern Cuttingcited by 8
  23. 2024OSSAR: Towards Open-Set Surgical Activity Recognition in Robot-assisted Surgerycited by 7
  24. 2020Dynamic Movement Primitives: Volumetric Obstacle Avoidance Using Dynamic Potential Functionscited by 5
  25. 2024Multi-objective Cross-task Learning via Goal-conditioned GPT-based Decision Transformers for Surgical Robot Task Automationcited by 4
  26. 2021Real-time landmark detection for precise endoscopic submucosal dissection via shape-aware relation networkcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 6; 2026: 8.

15 papers.

Sort:

By area:

  1. 2025SurgVLM: A Large Vision-Language Model and Systematic Evaluation Benchmark for Surgical Intelligencecited by 30
  2. 2024SurgSora: Object-Aware Diffusion Model for Controllable Surgical Video Generationcited by 12
  3. 2025MedGRPO: Multi-Task Reinforcement Learning for Heterogeneous Medical Video Understandingcited by 8
  4. 2026Photon: Speedup Volume Understanding with Efficient Multimodal Large Language Modelscited by 6
  5. 2026SurgMotion: A Video-Native Foundation Model for Universal Understanding of Surgical Videoscited by 4
  6. 2025Multimodal Graph Representation Learning for Robust Surgical Workflow Recognition with Adversarial Feature Disentanglementcited by 3
  7. 2026Surg-R1: A Hierarchical Reasoning Foundation Model for Scalable and Interpretable Surgical Decision Support with Multi-Center Clinical Validationcited by 2
  8. 2026Towards Next-Generation Healthcare: A Survey of Medical Embodied AI for Perception, Decision-Making, and Actioncited by 2
  9. 2025EndoARSS: Adapting Spatially-Aware Foundation Model for Efficient Activity Recognition and Semantic Segmentation in Endoscopic Surgerycited by 1
  10. 2025Geo-RepNet: Geometry-Aware Representation Learning for Surgical Phase Recognition in Endoscopic Submucosal Dissectioncited by 1
  11. 2026SurgSigma: A Spectrum of Large-Scale Multimodal Data and Foundation Models for Surgical Intelligencecited by 1
  12. 2025Can DeepSeek Reason Like a Surgeon? An Empirical Evaluation for Vision-Language Understanding in Robotic-Assisted Surgerycited by 0
3 more
  1. 2026SurgCoT: Advancing Spatiotemporal Reasoning in Surgical Videos through a Chain-of-Thought Benchmarkcited by 0
  2. 2026GeoCFNet: Geometry-Aware Confidence Field Network for Robot-Assisted Endoscopic Submucosal Dissectioncited by 0
  3. 2026RefineRank: Joint Box Refinement and Ranking for Surgical Spatio-Temporal Groundingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Challenging Vision-Language Models with Surgical Data: A New Dataset and Broad Benchmarking Study
  2. 2025EndoVLA: Dual-Phase Vision-Language-Action Model for Autonomous Tracking in Endoscopy
  3. 2025Learning dissection trajectories from expert surgical videos via imitation learning with equivariant diffusion
  4. 2025SurgiSR4K: A High-Resolution Endoscopic Video Dataset for Robotic-Assisted Minimally Invasive Procedures
  5. 2026EndoWAM: A Grounded World-Action Model for Generalizable Endoscopic Navigation
  6. 2025CPKD: Clinical Prior Knowledge-Constrained Diffusion Models for Surgical Phase Recognition in Endoscopic Submucosal Dissection
  7. 2025EndoChat: Grounded Multimodal Large Language Model for Endoscopic Surgery
  8. 2025Systematic Evaluation of Large Vision-Language Models for Surgical Artificial Intelligence