# Recognizing Surgical Phases Anywhere: Few-Shot Test-time Adaptation and Task-graph Guided Refinement

arXiv: https://arxiv.org/abs/2506.20254 · HTML: https://mlsanity.com/paper/2506.20254/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 30 references that are in the map.
- First posted on arXiv 25 Jun 2025.

## Builds on (most cited first, 25 of 25)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [MedCLIP: Contrastive Learning from Unpaired Medical Images and Text](https://mlsanity.com/paper/2210.10163/) (2022)
- [Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling](https://mlsanity.com/paper/2111.03930/) (2021)
- [EndoNet: A Deep Architecture for Recognition Tasks on Laparoscopic Videos](https://mlsanity.com/paper/1602.03012/) (2016)
- [Quilt-1M: One Million Image-Text Pairs for Histopathology](https://mlsanity.com/paper/2306.11207/) (2023)
- [MedKLIP: Medical Knowledge Enhanced Language-Image Pre-Training in Radiology](https://mlsanity.com/paper/2301.02228/) (2023)
- [TeCNO: Surgical Phase Recognition with Multi-Stage Temporal Convolutional Networks](https://mlsanity.com/paper/2003.10751/) (2020)
- [Surgical Data Science: Enabling Next-Generation Surgery](https://mlsanity.com/paper/1701.06482/) (2017)
- [AutoLaparo: A New Dataset of Integrated Multi-tasks for Image-guided Surgical Automation in Laparoscopic Hysterectomy](https://mlsanity.com/paper/2208.02049/) (2022)
- [A Foundation Language-Image Model of the Retina (FLAIR): Encoding Expert Knowledge in Text Supervision](https://mlsanity.com/paper/2308.07898/) (2023)
- [Learning Multi-modal Representations by Watching Hundreds of Surgical Video Lectures](https://mlsanity.com/paper/2307.15220/) (2023)
- [A Closer Look at the Few-Shot Adaptation of Large Vision-Language Models](https://mlsanity.com/paper/2312.12730/) (2023)
- [Dissecting Self-Supervised Learning Methods for Surgical Computer Vision](https://mlsanity.com/paper/2207.00449/) (2022)
- [HecVL: Hierarchical Video-Language Pretraining for Zero-shot Surgical Phase Recognition](https://mlsanity.com/paper/2405.10075/) (2024)
- [Challenges in Multi-centric Generalization: Phase and Step Recognition in Roux-en-Y Gastric Bypass Surgery](https://mlsanity.com/paper/2312.11250/) (2023)
- [Procedure-Aware Surgical Video-language Pretraining with Hierarchical Knowledge Augmentation](https://mlsanity.com/paper/2410.00263/) (2024)
- [Latent Graph Representations for Critical View of Safety Assessment](https://mlsanity.com/paper/2212.04155/) (2022)
- [OphCLIP: Hierarchical Retrieval-Augmented Learning for Ophthalmic Surgical Video-Language Pretraining](https://mlsanity.com/paper/2411.15421/) (2024)
- [Scaling up self-supervised learning for improved surgical foundation models](https://mlsanity.com/paper/2501.09436/) (2025)
- [Few-shot Adaptation of Medical Vision-Language Models](https://mlsanity.com/paper/2409.03868/) (2024)
- [Eye-gaze Guided Multi-modal Alignment for Medical Representation Learning](https://mlsanity.com/paper/2403.12416/) (2024)
- [ORacle: Large Vision-Language Models for Knowledge-Guided Holistic OR Domain Modeling](https://mlsanity.com/paper/2404.07031/) (2024)
- [VidLPRO: A underlineVideo-underlineLanguage underlinePre-training Framework for underlineRobotic and Laparoscopic Surgery](https://mlsanity.com/paper/2409.04732/) (2024)
- [Jumpstarting Surgical Computer Vision](https://mlsanity.com/paper/2312.05968/) (2023)
- [UltraAD: Fine-Grained Ultrasound Anomaly Classification via Few-Shot CLIP Adaptation](https://mlsanity.com/paper/2506.19694/) (2025)

## Built on by (most cited first, 4 of 4)

- [Where It Moves, It Matters: Referring Surgical Instrument Segmentation via Motion](https://mlsanity.com/paper/2601.12224/) (2026)
- [Geo-RepNet: Geometry-Aware Representation Learning for Surgical Phase Recognition in Endoscopic Submucosal Dissection](https://mlsanity.com/paper/2507.09294/) (2025)
- [Large-Small Model Collaboration for Zero-Shot Surgical Phase Recognition](https://mlsanity.com/paper/2608.22879/) (2026)
- [Morphological Decoupling-Based Skeletal Classification for Clinical Assessment of Malocclusion](https://mlsanity.com/paper/2609.09801/) (2026)

## Abstract (arXiv)

The complexity and diversity of surgical workflows, driven by heterogeneous operating room settings, institutional protocols, and anatomical variability, present a significant challenge in developing generalizable models for cross-institutional and cross-procedural surgical understanding. While recent surgical foundation models pretrained on large-scale vision-language data offer promising transferability, their zero-shot performance remains constrained by domain shifts, limiting their utility in unseen surgical environments. To address this, we introduce Surgical Phase Anywhere (SPA), a lightweight framework for versatile surgical workflow understanding that adapts foundation models to institutional settings with minimal annotation. SPA leverages few-shot spatial adaptation to align multi-modal embeddings with institution-specific surgical scenes and phases. It also ensures temporal consistency through diffusion modeling, which encodes task-graph priors derived from institutional procedure protocols. Finally, SPA employs dynamic test-time adaptation, exploiting the mutual agreement between multi-modal phase prediction streams to adapt the model to a given test video in a self-supervised manner, enhancing the reliability under test-time distribution shifts. SPA is a lightweight adaptation framework, allowing hospitals to rapidly customize phase recognition models by defining phases in natural language text, annotating a few images with the phase labels, and providing a task graph defining phase transitions. The experimental results show that the SPA framework achieves state-of-the-art performance in few-shot surgical phase recognition across multiple institutions and procedures, even outperforming full-shot models with 32-shot labeled data. Code is available at https://github.com/CAMMA-public/SPA