# Performance and Non-adversarial Robustness of the Segment Anything Model 2 in Surgical Video Segmentation

arXiv: https://arxiv.org/abs/2408.04098 · HTML: https://mlsanity.com/paper/2408.04098/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Aug 2024.

## Built on by (most cited first, 13 of 13)

- [Efficient Track Anything](https://mlsanity.com/paper/2411.18933/) (2024)
- [Unleashing the Potential of SAM2 for Biomedical Images and Videos: A Survey](https://mlsanity.com/paper/2408.12889/) (2024)
- [SAM2 for Image and Video Segmentation: A Comprehensive Survey](https://mlsanity.com/paper/2503.12781/) (2025)
- [Reasoning Segmentation for Images and Videos: A Survey](https://mlsanity.com/paper/2505.18816/) (2025)
- [When SAM2 Meets Video Camouflaged Object Segmentation: A Comprehensive Evaluation and Adaptation](https://mlsanity.com/paper/2409.18653/) (2024)
- [Memory-Augmented SAM2 for Training-Free Surgical Video Segmentation](https://mlsanity.com/paper/2507.09577/) (2025)
- [CamSAM2: Segment Anything Accurately in Camouflaged Videos](https://mlsanity.com/paper/2503.19730/) (2025)
- [Zero-shot capability of SAM-family models for bone segmentation in CT scans](https://mlsanity.com/paper/2411.08629/) (2024)
- [Towards Robust Algorithms for Surgical Phase Recognition via Digital Twin Representation](https://mlsanity.com/paper/2410.20026/) (2024)
- [LACOSTE: Exploiting stereo and temporal contexts for surgical instrument segmentation](https://mlsanity.com/paper/2409.09360/) (2024)
- [Beyond Rigid AI: Towards Natural Human-Machine Symbiosis for Interoperative Surgical Assistance](https://mlsanity.com/paper/2507.23088/) (2025)
- [Towards Better Ultrasound Video Segmentation Foundation Model: An Empirical study on SAM2 Finetuning from Data Perspective](https://mlsanity.com/paper/2511.05731/) (2025)
- [OASIC: Occlusion-Agnostic and Severity-Informed Classification](https://mlsanity.com/paper/2604.04012/) (2026)

## Abstract (arXiv)

Fully supervised deep learning (DL) models for surgical video segmentation have been shown to struggle with non-adversarial, real-world corruptions of image quality including smoke, bleeding, and low illumination. Foundation models for image segmentation, such as the segment anything model (SAM) that focuses on interactive prompt-based segmentation, move away from semantic classes and thus can be trained on larger and more diverse data, which offers outstanding zero-shot generalization with appropriate user prompts. Recently, building upon this success, SAM-2 has been proposed to further extend the zero-shot interactive segmentation capabilities from independent frame-by-frame to video segmentation. In this paper, we present a first experimental study evaluating SAM-2's performance on surgical video data. Leveraging the SegSTRONG-C MICCAI EndoVIS 2024 sub-challenge dataset, we assess SAM-2's effectiveness on uncorrupted endoscopic sequences and evaluate its non-adversarial robustness on videos with corrupted image quality simulating smoke, bleeding, and low brightness conditions under various prompt strategies. Our experiments demonstrate that SAM-2, in zero-shot manner, can achieve competitive or even superior performance compared to fully-supervised deep learning models on surgical video data, including under non-adversarial corruptions of image quality. Additionally, SAM-2 consistently outperforms the original SAM and its medical variants across all conditions. Finally, frame-sparse prompting can consistently outperform frame-wise prompting for SAM-2, suggesting that allowing SAM-2 to leverage its temporal modeling capabilities leads to more coherent and accurate segmentation compared to frequent prompting.