mlsanityData to 7 Oct 2026

Text Promptable Surgical Instrument Segmentation with Vision-Language Models

arXiv 2306.09244 ↗First posted 15 Jun 2023cs.CVcited by 27

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

In this paper, we propose a novel text promptable surgical instrument segmentation approach to overcome challenges associated with diversity and differentiation of surgical instruments in minimally invasive surgeries. We redefine the task as text promptable, thereby enabling a more nuanced comprehension of surgical instruments and adaptability to new instrument types. Inspired by recent advancements in vision-language models, we leverage pretrained image and text encoders as our model backbone and design a text promptable mask decoder consisting of attention- and convolution-based prompting schemes for surgical instrument segmentation prediction. Our model leverages multiple text prompts for each surgical instrument through a new mixture of prompts mechanism, resulting in enhanced segmentation performance. Additionally, we introduce a hard instrument area reinforcement module to improve image feature comprehension and segmentation precision. Extensive experiments on several surgical instrument segmentation datasets demonstrate our model's superior performance and promising generalization capability. To our knowledge, this is the first implementation of a promptable approach to surgical instrument segmentation, offering significant potential for practical application in the field of robotic-assisted surgery. Code is available at https://github.com/franciszzj/TP-SIS.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 44 of its 55 references that are in the map. First posted on arXiv 15 Jun 2023.

Builds on

44 of 55 references.

References · 44

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  7. 2023GPT-4 Technical Reportcited by 18,132
  8. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  9. 2023Segment Anythingcited by 8,158
  10. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  11. 2016Layer Normalizationcited by 5,579
  12. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
32 more
  1. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  2. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  3. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  4. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 990
  5. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  6. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 645
  7. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  8. 2021Image Segmentation Using Text and Image Promptscited by 509
  9. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 506
  10. 2021CRIS: CLIP-Driven Referring Image Segmentationcited by 353
  11. 2021LAVT: Language-Aware Vision Transformer for Referring Image Segmentationcited by 347
  12. 2022Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Modelcited by 317
  13. 2019Cross-Modal Self-Attention Network for Referring Image Segmentationcited by 263
  14. 2016Segmentation from Natural Language Expressionscited by 257
  15. 2021Vision-Language Transformer and Query Generation for Referring Segmentationcited by 239
  16. 2023CLIP-Driven Universal Model for Organ Segmentation and Tumor Detectioncited by 224
  17. 2021CLIP2Video: Mastering Video-Text Retrieval via Image CLIPcited by 223
  18. 2022ReSTR: Convolution-free Referring Image Segmentation Using Transformerscited by 112
  19. 2021Encoder Fusion Network with Co-Attention Embedding for Referring Image Segmentationcited by 111
  20. 2020CholecSeg8k: A Semantic Segmentation Dataset for Laparoscopic Cholecystectomy Based on Cholec80cited by 104
  21. 2021CLIP4Caption: CLIP for Video Captioncited by 88
  22. 2018Automatic Instrument Segmentation in Robot-Assisted Surgery Using Deep Learningcited by 81
  23. 2020ISINet: An Instance-Based Approach for Surgical Instrument Segmentationcited by 67
  24. 2019Incorporating Temporal Prior from Motion Flow for Instrument Segmentation in Minimally Invasive Surgery Videocited by 64
  25. 2023Zero-shot Referring Image Segmentation with Global-Local Context Featurescited by 61
  26. 2022Towards Understanding Mixture of Experts in Deep Learningcited by 52
  27. 2022Towards Robust Referring Image Segmentationcited by 49
  28. 2022TraSeTR: Track-to-Segment Transformer with Contrastive Query for Instance-level Instrument Segmentation in Robotic Surgerycited by 40
  29. 2023MATIS: Masked-Attention Transformers for Surgical Instrument Segmentationcited by 36
  30. 2022From Forks to Forceps: A New Framework for Instance Segmentation of Surgical Instrumentscited by 32
  31. 2023LoSh: Long-Short Text Joint Prediction Network for Referring Video Object Segmentationcited by 23
  32. 2020Robust Medical Instrument Segmentation Challenge 2019cited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 9; 2025: 9; 2026: 7.

27 papers.

Sort:

By area:

  1. 2024Image Segmentation in Foundation Model Era: A Surveycited by 35
  2. 2024OphCLIP: Hierarchical Retrieval-Augmented Learning for Ophthalmic Surgical Video-Language Pretrainingcited by 29
  3. 2025MM-OR: A Large Multimodal Operating Room Dataset for Semantic Understanding of High-Intensity Surgical Environmentscited by 28
  4. 2024CholecInstanceSeg: A Tool Instance Segmentation Dataset for Laparoscopic Surgerycited by 16
  5. 2025ReSurgSAM2: Referring Segment Anything in Surgical Video via Credible Long-term Trackingcited by 15
  6. 2025SurgVidLM: Towards Multi-grained Surgical Video Understanding with Large Language Modelcited by 13
  7. 2023VLPrompt: Vision-Language Prompting for Panoptic Scene Graph Generationcited by 12
  8. 2025Taming Vision-Language Models for Medical Image Analysis: A Comprehensive Reviewcited by 9
  9. 2023SurgicalPart-SAM: Part-to-Whole Collaborative Prompting for Surgical Instrument Segmentationcited by 8
  10. 2024Augmenting Efficient Real-time Surgical Instrument Segmentation in Video with Point Tracking and Segment Anythingcited by 7
  11. 2025SurgTPGS: Semantic 3D Surgical Scene Understanding with Text Promptable Gaussian Splattingcited by 7
  12. 2025Comparative validation of surgical phase recognition, instrument keypoint estimation, and instrument instance segmentation in endoscopy: Results of the PhaKIR 2024 challengecited by 6
15 more
  1. 2024Personalizing Federated Instrument Segmentation with Visual Trait Priors in Robotic Surgerycited by 2
  2. 2024Segment as You Wish -- Free-Form Language-Based Segmentation for Medical Imagescited by 2
  3. 2025SCOPE: Speech-guided COllaborative PErception Framework for Surgical Scene Segmentationcited by 1
  4. 2026An Approach to Enriching Surgical Video Datasets for Fine-Grained Spatial-Temporal Understanding of Vision-Language Modelscited by 1
  5. 2026Hierarchical Prototype-Memory Adaptation of SAM for Surgical Instrument Segmentationcited by 1
  6. 2024Unity in Diversity: Multi-expert Knowledge Confrontation and Collaboration for Generalizable Vehicle Re-identificationcited by 0
  7. 2024Bootstrapping Vision-language Models for Self-supervised Remote Physiological Measurementcited by 0
  8. 2024Rethinking Text-Promptable Surgical Instrument Segmentation with Robust Frameworkcited by 0
  9. 2025SurgPIS: Surgical-instrument-level Instances and Part-level Semantics for Weakly-supervised Part-aware Instance Segmentationcited by 0
  10. 2025ReMeDI: Refined Memory for Disambiguation of Identities with SAM3 in Surgical Segmentationcited by 0
  11. 2026VLM-Guided Iterative Refinement for Surgical Image Segmentation with Foundation Modelscited by 0
  12. 2026Toward Mask Annotation-Free Surgical Instrument Segmentation from Endoscopic Images Using Text-Prompted Segment Anything Model 3 (SAM3)cited by 0
  13. 2026Bootstrapping Vision-Language Model for Hysteroscopic Surgical Scene Segmentationcited by 0
  14. 2026SIRA: Reasoning-Aware Surgical Instrument Segmentation via Query-Anchored Alignmentcited by 0
  15. 2026LD-RSVIS: A Large-Scale and Diverse Benchmark for Referring Surgical Video Instrument Segmentationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026GroundedSurg: A Multi-Procedure Benchmark for Language-Conditioned Surgical Tool Segmentation
  2. 2025SurgMLLMBench: A Multimodal Large Language Model Benchmark Dataset for Surgical Scene Understanding
  3. 2025SurgXBench: Explainable Vision-Language Model Benchmark for Surgery
  4. 2024Surgical Text-to-Image Generation
  5. 2026TrajPred: Trajectory-Conditioned Joint Embedding Prediction for Surgical Instrument-Tissue Interaction Recognition in Vision-Language Models
  6. 2026Surgical Anatomy Recognition with Context Learning using Foundation Representations
  7. 2025SurgVLM: A Large Vision-Language Model and Systematic Evaluation Benchmark for Surgical Intelligence
  8. 2026Object Tokens as a Bridge Between Segmentation and Visual Question Answering in Robotic Surgery