# ReMeDI: Refined Memory for Disambiguation of Identities with SAM3 in Surgical Segmentation

arXiv: https://arxiv.org/abs/2512.16880 · HTML: https://mlsanity.com/paper/2512.16880/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 24 of its 24 references that are in the map.
- First posted on arXiv 18 Dec 2025.

## Builds on (most cited first, 24 of 24)

- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [SAM 3: Segment Anything with Concepts](https://mlsanity.com/paper/2511.16719/) (2025)
- [Perception Encoder: The best visual embeddings are not at the output of the network](https://mlsanity.com/paper/2504.13181/) (2025)
- [Personalize Segment Anything Model with One Shot](https://mlsanity.com/paper/2305.03048/) (2023)
- [Track Anything: Segment Anything Meets Videos](https://mlsanity.com/paper/2304.11968/) (2023)
- [2017 Robotic Instrument Segmentation Challenge](https://mlsanity.com/paper/1902.06426/) (2019)
- [2018 Robotic Scene Segmentation Challenge](https://mlsanity.com/paper/2001.11190/) (2020)
- [CholecSeg8k: A Semantic Segmentation Dataset for Laparoscopic Cholecystectomy Based on Cholec80](https://mlsanity.com/paper/2012.12453/) (2020)
- [SurgicalSAM: Efficient Class Promptable Surgical Instrument Segmentation](https://mlsanity.com/paper/2308.08746/) (2023)
- [SAMURAI: Adapting Segment Anything Model for Zero-Shot Visual Tracking with Motion-Aware Memory](https://mlsanity.com/paper/2411.11922/) (2024)
- [Automatic Instrument Segmentation in Robot-Assisted Surgery Using Deep Learning](https://mlsanity.com/paper/1803.01207/) (2018)
- [ISINet: An Instance-Based Approach for Surgical Instrument Segmentation](https://mlsanity.com/paper/2007.05533/) (2020)
- [A Distractor-Aware Memory for Visual Object Tracking with SAM2](https://mlsanity.com/paper/2411.17576/) (2024)
- [Surgical SAM 2: Real-time Segment Anything in Surgical Video by Efficient Frame Pruning](https://mlsanity.com/paper/2408.07931/) (2024)
- [MATIS: Masked-Attention Transformers for Surgical Instrument Segmentation](https://mlsanity.com/paper/2303.09514/) (2023)
- [From Forks to Forceps: A New Framework for Instance Segmentation of Surgical Instruments](https://mlsanity.com/paper/2211.16200/) (2022)
- [Text Promptable Surgical Instrument Segmentation with Vision-Language Models](https://mlsanity.com/paper/2306.09244/) (2023)
- [SurgicalPart-SAM: Part-to-Whole Collaborative Prompting for Surgical Instrument Segmentation](https://mlsanity.com/paper/2312.14481/) (2023)
- [SAMed-2: Selective Memory Enhanced Medical Segment Anything Model](https://mlsanity.com/paper/2507.03698/) (2025)
- [Memory-Augmented SAM2 for Training-Free Surgical Video Segmentation](https://mlsanity.com/paper/2507.09577/) (2025)
- [Adapting SAM for Surgical Instrument Tracking and Segmentation in Endoscopic Submucosal Dissection Videos](https://mlsanity.com/paper/2404.10640/) (2024)
- [Temporal Propagation of Asymmetric Feature Pyramid for Surgical Scene Segmentation](https://mlsanity.com/paper/2504.13440/) (2025)

## Abstract (arXiv)

Accurate surgical instrument segmentation in endoscopy is crucial for computer-assisted interventions, yet remains challenging due to frequent occlusions, rapid motion, and long-term instrument re-entry. While SAM3 provides a powerful spatio-temporal framework for video object segmentation, its performance in surgical scenes is limited by indiscriminate memory updates, fixed memory capacity, and weak identity recovery after occlusions. We propose ReMeDI-SAM3, a training-free extension of SAM3, that addresses these limitations through three components: (i) relevance-aware memory filtering with a dedicated occlusion-aware memory for storing pre-occlusion frames, (ii) a piecewise interpolation scheme that expands effective memory capacity, and (iii) a feature-based re-identification module with temporal voting for reliable post-occlusion identity disambiguation. Together, these components mitigate error accumulation and enable reliable recovery after occlusions. Evaluations on EndoVis17, EndoVis18 and CholecSeg8k under a zero-shot setting show mcIoU improvements of around 5.8%, 8%, and 2% respectively, over vanilla SAM3, outperforming even prior training-based approaches.