# One to Many: Adaptive Instrument Segmentation via Meta Learning and Dynamic Online Adaptation in Robotic Surgical Video

arXiv: https://arxiv.org/abs/2103.12988 · HTML: https://mlsanity.com/paper/2103.12988/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 34 references that are in the map.
- First posted on arXiv 24 Mar 2021.

## Builds on (most cited first, 28 of 28)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Few-Shot Learning with Graph Neural Networks](https://mlsanity.com/paper/1711.04043/) (2017)
- [Meta-Transfer Learning for Few-Shot Learning](https://mlsanity.com/paper/1812.02391/) (2018)
- [Recasting Gradient-Based Meta-Learning as Hierarchical Bayes](https://mlsanity.com/paper/1801.08930/) (2018)
- [2017 Robotic Instrument Segmentation Challenge](https://mlsanity.com/paper/1902.06426/) (2019)
- [Task-Agnostic Meta-Learning for Few-shot Learning](https://mlsanity.com/paper/1805.07722/) (2018)
- [2018 Robotic Scene Segmentation Challenge](https://mlsanity.com/paper/2001.11190/) (2020)
- [Online Adaptation of Convolutional Neural Networks for Video Object Segmentation](https://mlsanity.com/paper/1706.09364/) (2017)
- [Automatic Instrument Segmentation in Robot-Assisted Surgery Using Deep Learning](https://mlsanity.com/paper/1803.01207/) (2018)
- [Incorporating Temporal Prior from Motion Flow for Instrument Segmentation in Minimally Invasive Surgery Video](https://mlsanity.com/paper/1907.07899/) (2019)
- [Learning to Adapt for Stereo](https://mlsanity.com/paper/1904.02957/) (2019)
- [Domain Adaptive Dialog Generation via Meta Learning](https://mlsanity.com/paper/1906.03520/) (2019)
- [Generating large labeled data sets for laparoscopic image processing tasks using unpaired image-to-image translation](https://mlsanity.com/paper/1907.02882/) (2019)
- [Blending-target Domain Adaptation by Adversarial Meta-Adaptation Networks](https://mlsanity.com/paper/1907.03389/) (2019)
- [Learning Motion Flows for Semi-supervised Instrument Segmentation from Robotic Surgical Video](https://mlsanity.com/paper/2007.02501/) (2020)
- [Synthetic and Real Inputs for Tool Segmentation in Robotic Surgery](https://mlsanity.com/paper/2007.09107/) (2020)
- [Real-Time Instrument Segmentation in Robotic Surgery using Auxiliary Supervised Deep Adversarial Learning](https://mlsanity.com/paper/2007.11319/) (2020)
- [FOAL: Fast Online Adaptive Learning for Cardiac Motion Estimation](https://mlsanity.com/paper/2003.04492/) (2020)
- [Endo-Sim2Real: Consistency learning-based domain adaptation for instrument segmentation](https://mlsanity.com/paper/2007.11514/) (2020)
- [Surgical Activity Recognition in Robot-Assisted Radical Prostatectomy using Deep Learning](https://mlsanity.com/paper/1806.00466/) (2018)
- [Learning Where to Look While Tracking Instruments in Robot-assisted Surgery](https://mlsanity.com/paper/1907.00214/) (2019)
- [AP-MTL: Attention Pruned Multi-task Learning Model for Real-time Instrument Detection and Segmentation in Robot-assisted Surgery](https://mlsanity.com/paper/2003.04769/) (2020)
- [Self-Supervised Surgical Tool Segmentation using Kinematic Information](https://mlsanity.com/paper/1902.04810/) (2019)
- [Attention-Guided Lightweight Network for Real-Time Segmentation of Robotic Surgical Instruments](https://mlsanity.com/paper/1910.11109/) (2019)
- [Future Frame Prediction for Robot-assisted Surgery](https://mlsanity.com/paper/2103.10308/) (2021)
- [Online Adaptation through Meta-Learning for Stereo Depth Estimation](https://mlsanity.com/paper/1904.08462/) (2019)

## Built on by (most cited first, 14 of 14)

- [TraSeTR: Track-to-Segment Transformer with Contrastive Query for Instance-level Instrument Segmentation in Robotic Surgery](https://mlsanity.com/paper/2202.08453/) (2022)
- [MATIS: Masked-Attention Transformers for Surgical Instrument Segmentation](https://mlsanity.com/paper/2303.09514/) (2023)
- [Methods and datasets for segmentation of minimally invasive surgical instruments in endoscopic images and videos: A review of the state of the art](https://mlsanity.com/paper/2304.13014/) (2023)
- [CaRTS: Causality-driven Robot Tool Segmentation from Vision and Kinematics Data](https://mlsanity.com/paper/2203.09475/) (2022)
- [Rethinking Causality-driven Robot Tool Segmentation with Temporal Constraints](https://mlsanity.com/paper/2212.00072/) (2022)
- [Harnessing Meta-Learning for Improving Full-Frame Video Stabilization](https://mlsanity.com/paper/2403.03662/) (2024)
- [Systematic Evaluation and Guidelines for Segment Anything Model in Surgical Video Analysis](https://mlsanity.com/paper/2501.00525/) (2024)
- [One-shot skill assessment in high-stakes domains with limited data via meta learning](https://mlsanity.com/paper/2301.00812/) (2022)
- [Real-time landmark detection for precise endoscopic submucosal dissection via shape-aware relation network](https://mlsanity.com/paper/2111.04733/) (2021)
- [A comprehensive survey on recent deep learning-based methods applied to surgical data](https://mlsanity.com/paper/2209.01435/) (2022)
- [Personalizing Federated Instrument Segmentation with Visual Trait Priors in Robotic Surgery](https://mlsanity.com/paper/2408.03208/) (2024)
- [VideoSAM: Open-World Video Segmentation](https://mlsanity.com/paper/2410.08781/) (2024)
- [Harnessing Meta-Learning for Controllable Full-Frame Video Stabilization](https://mlsanity.com/paper/2508.18859/) (2025)
- [Visual-Kinematics Graph Learning for Procedure-agnostic Instrument Tip Segmentation in Robotic Surgeries](https://mlsanity.com/paper/2309.00957/) (2023)

## Abstract (arXiv)

Surgical instrument segmentation in robot-assisted surgery (RAS) - especially that using learning-based models - relies on the assumption that training and testing videos are sampled from the same domain. However, it is impractical and expensive to collect and annotate sufficient data from every new domain. To greatly increase the label efficiency, we explore a new problem, i.e., adaptive instrument segmentation, which is to effectively adapt one source model to new robotic surgical videos from multiple target domains, only given the annotated instruments in the first frame. We propose MDAL, a meta-learning based dynamic online adaptive learning scheme with a two-stage framework to fast adapt the model parameters on the first frame and partial subsequent frames while predicting the results. MDAL learns the general knowledge of instruments and the fast adaptation ability through the video-specific meta-learning paradigm. The added gradient gate excludes the noisy supervision from pseudo masks for dynamic online adaptation on target videos. We demonstrate empirically that MDAL outperforms other state-of-the-art methods on two datasets (including a real-world RAS dataset). The promising performance on ex-vivo scenes also benefits the downstream tasks such as robot-assisted suturing and camera control.