# Label-supervised surgical instrument segmentation using temporal equivariance and semantic continuity

arXiv: https://arxiv.org/abs/2403.09551 · HTML: https://mlsanity.com/paper/2403.09551/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 29 references that are in the map.
- First posted on arXiv 14 Mar 2024.

## Builds on (most cited first, 22 of 22)

- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [EndoNet: A Deep Architecture for Recognition Tasks on Laparoscopic Videos](https://mlsanity.com/paper/1602.03012/) (2016)
- [Multi-class Token Transformer for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2203.02891/) (2022)
- [ISINet: An Instance-Based Approach for Surgical Instrument Segmentation](https://mlsanity.com/paper/2007.05533/) (2020)
- [Token Contrast for Weakly-Supervised Semantic Segmentation](https://mlsanity.com/paper/2303.01267/) (2023)
- [Incorporating Temporal Prior from Motion Flow for Instrument Segmentation in Minimally Invasive Surgery Video](https://mlsanity.com/paper/1907.07899/) (2019)
- [TraSeTR: Track-to-Segment Transformer with Contrastive Query for Instance-level Instrument Segmentation in Robotic Surgery](https://mlsanity.com/paper/2202.08453/) (2022)
- [MATIS: Masked-Attention Transformers for Surgical Instrument Segmentation](https://mlsanity.com/paper/2303.09514/) (2023)
- [Weakly Supervised Convolutional LSTM Approach for Tool Tracking in Laparoscopic Videos](https://mlsanity.com/paper/1812.01366/) (2018)
- [Time Does Tell: Self-Supervised Time-Tuning of Dense Image Representations](https://mlsanity.com/paper/2308.11796/) (2023)
- [From Forks to Forceps: A New Framework for Instance Segmentation of Surgical Instruments](https://mlsanity.com/paper/2211.16200/) (2022)
- [U-NetPlus: A Modified Encoder-Decoder U-Net Architecture for Semantic and Instance Segmentation of Surgical Instrument](https://mlsanity.com/paper/1902.08994/) (2019)
- [MCTformer+: Multi-Class Token Transformer for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2308.03005/) (2023)
- [FUN-SIS: a Fully UNsupervised approach for Surgical Instrument Segmentation](https://mlsanity.com/paper/2202.08141/) (2022)
- [Learning Where to Look While Tracking Instruments in Robot-assisted Surgery](https://mlsanity.com/paper/1907.00214/) (2019)
- [Unsupervised Surgical Instrument Segmentation via Anchor Generation and Semantic Diffusion](https://mlsanity.com/paper/2008.11946/) (2020)
- [OR-UNet: an Optimized Robust Residual U-Net for Instrument Segmentation in Endoscopic Images](https://mlsanity.com/paper/2004.12668/) (2020)
- [Simulation-to-Real domain adaptation with teacher-student learning for endoscopic instrument segmentation](https://mlsanity.com/paper/2103.01593/) (2021)
- [MURPHY: Relations Matter in Surgical Workflow Analysis](https://mlsanity.com/paper/2212.12719/) (2022)

## Abstract (arXiv)

For robotic surgical videos, instrument presence annotations are typically recorded with video streams, which offering the potential to reduce the manually annotated costs for segmentation. However, weakly supervised surgical instrument segmentation with only instrument presence labels has been rarely explored in surgical domain due to the highly under-constrained challenges. Temporal properties can enhance representation learning by capturing sequential dependencies and patterns over time even in incomplete supervision situations. From this, we take the inherent temporal attributes of surgical video into account and extend a two-stage weakly supervised segmentation paradigm from different perspectives. Firstly, we make temporal equivariance constraint to enhance pixel-wise temporal consistency between adjacent features. Secondly, we constrain class-aware semantic continuity between global and local regions across temporal dimension. Finally, we generate temporal-enhanced pseudo masks from consecutive frames to suppress irrelevant regions. Extensive experiments are validated on two surgical video datasets, including one cholecystectomy surgery benchmark and one real robotic left lateral segment liver surgery dataset. We annotate instance-wise instrument labels with fixed time-steps which are double checked by a clinician with 3-years experience to evaluate segmentation results. Experimental results demonstrate the promising performances of our method, which consistently achieves comparable or favorable results with previous state-of-the-art approaches.