# MATIS: Masked-Attention Transformers for Surgical Instrument Segmentation

arXiv: https://arxiv.org/abs/2303.09514 · HTML: https://mlsanity.com/paper/2303.09514/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 38 references that are in the map.
- First posted on arXiv 16 Mar 2023.

## Builds on (most cited first, 28 of 28)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [Per-Pixel Classification is Not All You Need for Semantic Segmentation](https://mlsanity.com/paper/2107.06278/) (2021)
- [Video Swin Transformer](https://mlsanity.com/paper/2106.13230/) (2021)
- [Multiscale Vision Transformers](https://mlsanity.com/paper/2104.11227/) (2021)
- [2017 Robotic Instrument Segmentation Challenge](https://mlsanity.com/paper/1902.06426/) (2019)
- [2018 Robotic Scene Segmentation Challenge](https://mlsanity.com/paper/2001.11190/) (2020)
- [Automatic Instrument Segmentation in Robot-Assisted Surgery Using Deep Learning](https://mlsanity.com/paper/1803.01207/) (2018)
- [ISINet: An Instance-Based Approach for Surgical Instrument Segmentation](https://mlsanity.com/paper/2007.05533/) (2020)
- [Incorporating Temporal Prior from Motion Flow for Instrument Segmentation in Minimally Invasive Surgery Video](https://mlsanity.com/paper/1907.07899/) (2019)
- [TraSeTR: Track-to-Segment Transformer with Contrastive Query for Instance-level Instrument Segmentation in Robotic Surgery](https://mlsanity.com/paper/2202.08453/) (2022)
- [Comparative evaluation of instrument segmentation and tracking methods in minimally invasive surgery](https://mlsanity.com/paper/1805.02475/) (2018)
- [Learning Motion Flows for Semi-supervised Instrument Segmentation from Robotic Surgical Video](https://mlsanity.com/paper/2007.02501/) (2020)
- [Towards Holistic Surgical Scene Understanding](https://mlsanity.com/paper/2212.04582/) (2022)
- [ToolNet: Holistically-Nested Real-Time Segmentation of Robotic Surgical Tools](https://mlsanity.com/paper/1706.08126/) (2017)
- [Exploring Intra- and Inter-Video Relation for Surgical Semantic Scene Segmentation](https://mlsanity.com/paper/2203.15251/) (2022)
- [Synthetic and Real Inputs for Tool Segmentation in Robotic Surgery](https://mlsanity.com/paper/2007.09107/) (2020)
- [Concurrent Segmentation and Localization for Tracking of Surgical Instruments](https://mlsanity.com/paper/1703.10701/) (2017)
- [U-NetPlus: A Modified Encoder-Decoder U-Net Architecture for Semantic and Instance Segmentation of Surgical Instrument](https://mlsanity.com/paper/1902.08994/) (2019)
- [Learning Where to Look While Tracking Instruments in Robot-assisted Surgery](https://mlsanity.com/paper/1907.00214/) (2019)
- [Self-Supervised Surgical Tool Segmentation using Kinematic Information](https://mlsanity.com/paper/1902.04810/) (2019)
- [One to Many: Adaptive Instrument Segmentation via Meta Learning and Dynamic Online Adaptation in Robotic Surgical Video](https://mlsanity.com/paper/2103.12988/) (2021)

## Built on by (most cited first, 30 of 36)

- [SurgicalSAM: Efficient Class Promptable Surgical Instrument Segmentation](https://mlsanity.com/paper/2308.08746/) (2023)
- [SAR-RARP50: Segmentation of surgical instrumentation and Action Recognition on Robot-Assisted Radical Prostatectomy Challenge](https://mlsanity.com/paper/2401.00496/) (2023)
- [Pixel-Wise Recognition for Holistic Surgical Scene Understanding](https://mlsanity.com/paper/2401.11174/) (2024)
- [Surgical SAM 2: Real-time Segment Anything in Surgical Video by Efficient Frame Pruning](https://mlsanity.com/paper/2408.07931/) (2024)
- [Text Promptable Surgical Instrument Segmentation with Vision-Language Models](https://mlsanity.com/paper/2306.09244/) (2023)
- [SurgicalPart-SAM: Part-to-Whole Collaborative Prompting for Surgical Instrument Segmentation](https://mlsanity.com/paper/2312.14481/) (2023)
- [Memory-Augmented SAM2 for Training-Free Surgical Video Segmentation](https://mlsanity.com/paper/2507.09577/) (2025)
- [Comparative validation of surgical phase recognition, instrument keypoint estimation, and instrument instance segmentation in endoscopy: Results of the PhaKIR 2024 challenge](https://mlsanity.com/paper/2507.16559/) (2025)
- [Segment Anything for Videos: A Systematic Survey](https://mlsanity.com/paper/2408.08315/) (2024)
- [Tracking Everything in Robotic-Assisted Surgery](https://mlsanity.com/paper/2409.19821/) (2024)
- [Surgical Action Planning with Large Language Models](https://mlsanity.com/paper/2503.18296/) (2025)
- [ProstaTD: Bridging Surgical Triplet from Classification to Fully Supervised Detection](https://mlsanity.com/paper/2506.01130/) (2025)
- [From Articulated Kinematics to Routed Visual Control for Action-Conditioned Surgical Video Generation](https://mlsanity.com/paper/2605.08712/) (2026)
- [Personalizing Federated Instrument Segmentation with Visual Trait Priors in Robotic Surgery](https://mlsanity.com/paper/2408.03208/) (2024)
- [Phase-Informed Tool Segmentation for Manual Small-Incision Cataract Surgery](https://mlsanity.com/paper/2411.16794/) (2024)
- [Cataract-LMM Large-Scale Multi-Source Multi-Task Benchmark for Deep Learning in Surgical Video Analysis](https://mlsanity.com/paper/2510.16371/) (2025)
- [Where It Moves, It Matters: Referring Surgical Instrument Segmentation via Motion](https://mlsanity.com/paper/2601.12224/) (2026)
- [Recognize Any Surgical Object: Unleashing the Power of Weakly-Supervised Data](https://mlsanity.com/paper/2501.15326/) (2025)
- [Image Synthesis with Class-Aware Semantic Diffusion Models for Surgical Scene Segmentation](https://mlsanity.com/paper/2410.23962/) (2024)
- [TSMS-SAM2: Multi-scale Temporal Sampling Augmentation and Memory-Splitting Pruning for Promptable Video Object Segmentation and Tracking in Surgical Scenarios](https://mlsanity.com/paper/2508.05829/) (2025)
- [SurgSLOT: Segment Anything in Surgical Videos via Semantic Long-term Tracking](https://mlsanity.com/paper/2511.16618/) (2025)
- [An Approach to Enriching Surgical Video Datasets for Fine-Grained Spatial-Temporal Understanding of Vision-Language Models](https://mlsanity.com/paper/2604.00784/) (2026)
- [UniSurgSAM: A Unified Promptable Model for Reliable Surgical Video Segmentation](https://mlsanity.com/paper/2604.03645/) (2026)
- [Hierarchical Prototype-Memory Adaptation of SAM for Surgical Instrument Segmentation](https://mlsanity.com/paper/2608.24541/) (2026)
- [Label-supervised surgical instrument segmentation using temporal equivariance and semantic continuity](https://mlsanity.com/paper/2403.09551/) (2024)
- [LACOSTE: Exploiting stereo and temporal contexts for surgical instrument segmentation](https://mlsanity.com/paper/2409.09360/) (2024)
- [Rethinking Text-Promptable Surgical Instrument Segmentation with Robust Framework](https://mlsanity.com/paper/2411.12199/) (2024)
- [CLIP-RL: Surgical Scene Segmentation Using Contrastive Language-Vision Pretraining & Reinforcement Learning](https://mlsanity.com/paper/2507.04317/) (2025)
- [SurgPIS: Surgical-instrument-level Instances and Part-level Semantics for Weakly-supervised Part-aware Instance Segmentation](https://mlsanity.com/paper/2507.19592/) (2025)
- [FASL-Seg: Anatomy and Tool Segmentation of Surgical Scenes](https://mlsanity.com/paper/2509.06159/) (2025)

## Abstract (arXiv)

We propose Masked-Attention Transformers for Surgical Instrument Segmentation (MATIS), a two-stage, fully transformer-based method that leverages modern pixel-wise attention mechanisms for instrument segmentation. MATIS exploits the instance-level nature of the task by employing a masked attention module that generates and classifies a set of fine instrument region proposals. Our method incorporates long-term video-level information through video transformers to improve temporal consistency and enhance mask classification. We validate our approach in the two standard public benchmarks, Endovis 2017 and Endovis 2018. Our experiments demonstrate that MATIS' per-frame baseline outperforms previous state-of-the-art methods and that including our temporal consistency module boosts our model's performance further.