# Memory-Augmented SAM2 for Training-Free Surgical Video Segmentation

arXiv: https://arxiv.org/abs/2507.09577 · HTML: https://mlsanity.com/paper/2507.09577/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 28 references that are in the map.
- First posted on arXiv 13 Jul 2025.

## Builds on (most cited first, 20 of 20)

- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Medical SAM Adapter: Adapting Segment Anything Model for Medical Image Segmentation](https://mlsanity.com/paper/2304.12620/) (2023)
- [Personalize Segment Anything Model with One Shot](https://mlsanity.com/paper/2305.03048/) (2023)
- [Track Anything: Segment Anything Meets Videos](https://mlsanity.com/paper/2304.11968/) (2023)
- [2017 Robotic Instrument Segmentation Challenge](https://mlsanity.com/paper/1902.06426/) (2019)
- [Medical SAM 2: Segment medical images as video via Segment Anything Model 2](https://mlsanity.com/paper/2408.00874/) (2024)
- [2018 Robotic Scene Segmentation Challenge](https://mlsanity.com/paper/2001.11190/) (2020)
- [Empowering Things with Intelligence: A Survey of the Progress, Challenges, and Opportunities in Artificial Intelligence of Things](https://mlsanity.com/paper/2011.08612/) (2020)
- [SurgicalSAM: Efficient Class Promptable Surgical Instrument Segmentation](https://mlsanity.com/paper/2308.08746/) (2023)
- [Automatic Instrument Segmentation in Robot-Assisted Surgery Using Deep Learning](https://mlsanity.com/paper/1803.01207/) (2018)
- [SAM2Long: Enhancing SAM 2 for Long Video Segmentation with a Training-Free Memory Tree](https://mlsanity.com/paper/2410.16268/) (2024)
- [ISINet: An Instance-Based Approach for Surgical Instrument Segmentation](https://mlsanity.com/paper/2007.05533/) (2020)
- [A Distractor-Aware Memory for Visual Object Tracking with SAM2](https://mlsanity.com/paper/2411.17576/) (2024)
- [Surgical SAM 2: Real-time Segment Anything in Surgical Video by Efficient Frame Pruning](https://mlsanity.com/paper/2408.07931/) (2024)
- [MATIS: Masked-Attention Transformers for Surgical Instrument Segmentation](https://mlsanity.com/paper/2303.09514/) (2023)
- [SAM 2 in Robotic Surgery: An Empirical Evaluation for Robustness and Generalization in Surgical Video Segmentation](https://mlsanity.com/paper/2408.04593/) (2024)
- [SAM-LAD: Segment Anything Model Meets Zero-Shot Logic Anomaly Detection](https://mlsanity.com/paper/2406.00625/) (2024)
- [Biomedical SAM 2: Segment Anything in Biomedical Images and Videos](https://mlsanity.com/paper/2408.03286/) (2024)
- [Performance and Non-adversarial Robustness of the Segment Anything Model 2 in Surgical Video Segmentation](https://mlsanity.com/paper/2408.04098/) (2024)
- [Leaf Only SAM: A Segment Anything Pipeline for Zero-Shot Automated Leaf Segmentation](https://mlsanity.com/paper/2305.09418/) (2023)

## Built on by (most cited first, 8 of 8)

- [SurgSLOT: Segment Anything in Surgical Videos via Semantic Long-term Tracking](https://mlsanity.com/paper/2511.16618/) (2025)
- [Detector-in-the-Loop Tracking: Active Memory Rectification for Stable Glottic Opening Localization](https://mlsanity.com/paper/2602.19380/) (2026)
- [Hierarchical Prototype-Memory Adaptation of SAM for Surgical Instrument Segmentation](https://mlsanity.com/paper/2608.24541/) (2026)
- [Synergistic Modality-and-Slice Memory Framework for Cross-Modal 3D Brain Tumor Segmentation](https://mlsanity.com/paper/2510.10679/) (2025)
- [ReMeDI: Refined Memory for Disambiguation of Identities with SAM3 in Surgical Segmentation](https://mlsanity.com/paper/2512.16880/) (2025)
- [EchoPilot: Training-Free Ultrasound Video Segmentation via Scale-Space Semantic Prompting and Reliability-Gated Memory](https://mlsanity.com/paper/2605.25944/) (2026)
- [Temporally Consistent Label Interpolation for Robust Surgical Multi-Task Learning under Challenging Conditions](https://mlsanity.com/paper/2606.26634/) (2026)
- [ReGround-Surg: Reliability-Guided Anchor Grounding for Referring Surgical Video Segmentation](https://mlsanity.com/paper/2608.24671/) (2026)

## Abstract (arXiv)

Surgical video segmentation is a critical task in computer-assisted surgery, essential for enhancing surgical quality and patient outcomes. Recently, the Segment Anything Model 2 (SAM2) framework has demonstrated remarkable advancements in both image and video segmentation. However, the inherent limitations of SAM2's greedy selection memory design are amplified by the unique properties of surgical videos-rapid instrument movement, frequent occlusion, and complex instrument-tissue interaction-resulting in diminished performance in the segmentation of complex, long videos. To address these challenges, we introduce Memory Augmented (MA)-SAM2, a training-free video object segmentation strategy, featuring novel context-aware and occlusion-resilient memory models. MA-SAM2 exhibits strong robustness against occlusions and interactions arising from complex instrument movements while maintaining accuracy in segmenting objects throughout videos. Employing a multi-target, single-loop, one-prompt inference further enhances the efficiency of the tracking process in multi-instrument videos. Without introducing any additional parameters or requiring further training, MA-SAM2 achieved performance improvements of 4.36% and 6.1% over SAM2 on the EndoVis2017 and EndoVis2018 datasets, respectively, demonstrating its potential for practical surgical applications.