# SurgSLOT: Segment Anything in Surgical Videos via Semantic Long-term Tracking arXiv: https://arxiv.org/abs/2511.16618 ยท HTML: https://mlsanity.com/paper/2511.16618/ ## Key facts - Cited by 1 papers in the map (data through 14 Oct 2026). - Builds on 29 of its 45 references that are in the map. - First posted on arXiv 20 Nov 2025. ## Builds on (most cited first, 29 of 29) - [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021) - [Language-driven Semantic Segmentation](https://mlsanity.com/paper/2201.03546/) (2022) - [Kvasir-SEG: A Segmented Polyp Dataset](https://mlsanity.com/paper/1911.07069/) (2019) - [Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP](https://mlsanity.com/paper/2210.04150/) (2022) - [PANet: Few-Shot Image Semantic Segmentation with Prototype Alignment](https://mlsanity.com/paper/1908.06391/) (2019) - [XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Model](https://mlsanity.com/paper/2207.07115/) (2022) - [Prior Guided Feature Enrichment Network for Few-Shot Segmentation](https://mlsanity.com/paper/2008.01449/) (2020) - [Putting the Object Back into Video Object Segmentation](https://mlsanity.com/paper/2310.12982/) (2023) - [2017 Robotic Instrument Segmentation Challenge](https://mlsanity.com/paper/1902.06426/) (2019) - [Reviving Iterative Training with Mask Guidance for Interactive Segmentation](https://mlsanity.com/paper/2102.06583/) (2021) - [2018 Robotic Scene Segmentation Challenge](https://mlsanity.com/paper/2001.11190/) (2020) - [CholecSeg8k: A Semantic Segmentation Dataset for Laparoscopic Cholecystectomy Based on Cholec80](https://mlsanity.com/paper/2012.12453/) (2020) - [Surgical Data Science -- from Concepts toward Clinical Translation](https://mlsanity.com/paper/2011.02284/) (2020) - [SurgicalSAM: Efficient Class Promptable Surgical Instrument Segmentation](https://mlsanity.com/paper/2308.08746/) (2023) - [SAMURAI: Adapting Segment Anything Model for Zero-Shot Visual Tracking with Motion-Aware Memory](https://mlsanity.com/paper/2411.11922/) (2024) - [AutoLaparo: A New Dataset of Integrated Multi-tasks for Image-guided Surgical Automation in Laparoscopic Hysterectomy](https://mlsanity.com/paper/2208.02049/) (2022) - [SAM2Long: Enhancing SAM 2 for Long Video Segmentation with a Training-Free Memory Tree](https://mlsanity.com/paper/2410.16268/) (2024) - [A Distractor-Aware Memory for Visual Object Tracking with SAM2](https://mlsanity.com/paper/2411.17576/) (2024) - [NeoUNet: Towards accurate colon polyp segmentation and neoplasm detection](https://mlsanity.com/paper/2107.05023/) (2021) - [Pixel-Wise Recognition for Holistic Surgical Scene Understanding](https://mlsanity.com/paper/2401.11174/) (2024) - [ScribblePrompt: Fast and Flexible Interactive Segmentation for Any Biomedical Image](https://mlsanity.com/paper/2312.07381/) (2023) - [Surgical SAM 2: Real-time Segment Anything in Surgical Video by Efficient Frame Pruning](https://mlsanity.com/paper/2408.07931/) (2024) - [TraSeTR: Track-to-Segment Transformer with Contrastive Query for Instance-level Instrument Segmentation in Robotic Surgery](https://mlsanity.com/paper/2202.08453/) (2022) - [MATIS: Masked-Attention Transformers for Surgical Instrument Segmentation](https://mlsanity.com/paper/2303.09514/) (2023) - [CholecInstanceSeg: A Tool Instance Segmentation Dataset for Laparoscopic Surgery](https://mlsanity.com/paper/2406.16039/) (2024) - [ReSurgSAM2: Referring Segment Anything in Surgical Video via Credible Long-term Tracking](https://mlsanity.com/paper/2505.08581/) (2025) - [Deep Learning for Surgical Instrument Recognition and Segmentation in Robotic-Assisted Surgeries: A Systematic Review](https://mlsanity.com/paper/2410.07269/) (2024) - [Memory-Augmented SAM2 for Training-Free Surgical Video Segmentation](https://mlsanity.com/paper/2507.09577/) (2025) - [Spatio-Temporal Representation Decoupling and Enhancement for Federated Instrument Segmentation in Surgical Videos](https://mlsanity.com/paper/2506.23759/) (2025) ## Built on by (most cited first, 1 of 1) - [Training LLMs with Reinforcement Learning over Digital Twin Representations for Reasoning-Intensive Surgical VideoQA](https://mlsanity.com/paper/2606.17279/) (2026) ## Abstract (arXiv) Surgical scene understanding demands temporally consistent tracking of instruments and tissues. For clinical use, such tracking should generalize to new centers and procedure types, yet retraining for each of them is costly and not scalable. Interactive video object segmentation offers a way toward this generalization: the target is specified at inference by a first-frame visual prompt, so a model can generalize to unseen categories and new scenarios without retraining. However, training such a generalizable model demands spatio-temporal masklet annotations at a scale and procedural diversity that existing surgical benchmarks lack. We fill this gap with iSurg, the largest surgical segmentation benchmark to our knowledge, spanning six procedure types with over 170k frames, 410k object masks, and 2.4k masklets, including an in-house clinical dataset of four 30-minute videos. Yet tracking over such long videos remains challenging: the target needs to be re-identified among visually similar objects after long absences, while long-term memory has to judge whether each stored frame genuinely depicts the target, both demanding a stable object-level semantic identity. To this end, we propose SurgSLOT, a surgical segmentation generalist that segments any prompted target through two coupled modules built on this semantic identity: Temporal Semantic Learning learns it for re-identification, and Semantic-driven Long-term Memory reuses it to select reliable memory frames, suppressing identity drift over long procedures. On the SAM2 and SAM3 backbones, SurgSLOT reaches 81.0 and 82.8 Macro Average J&F under cross-dataset evaluation, surpassing their fine-tuned counterparts by 5.1 and 5.3 points and transferring zero-shot to an unseen procedure type and unseen object categories, with the SAM2 version running in real time at 68 FPS.