# Weakly Supervised Temporal Convolutional Networks for Fine-grained Surgical Activity Recognition

arXiv: https://arxiv.org/abs/2302.10834 · HTML: https://mlsanity.com/paper/2302.10834/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 44 references that are in the map.
- First posted on arXiv 21 Feb 2023.

## Builds on (most cited first, 26 of 26)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecture](https://mlsanity.com/paper/1411.4734/) (2014)
- [MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentation](https://mlsanity.com/paper/1903.01945/) (2019)
- [EndoNet: A Deep Architecture for Recognition Tasks on Laparoscopic Videos](https://mlsanity.com/paper/1602.03012/) (2016)
- [TeCNO: Surgical Phase Recognition with Multi-Stage Temporal Convolutional Networks](https://mlsanity.com/paper/2003.10751/) (2020)
- [Surgical Data Science: Enabling Next-Generation Surgery](https://mlsanity.com/paper/1701.06482/) (2017)
- [Multi-Task Recurrent Convolutional Network with Correlation Loss for Surgical Video Analysis](https://mlsanity.com/paper/1907.06099/) (2019)
- [Recognition of Instrument-Tissue Interactions in Endoscopic Videos via Action Triplets](https://mlsanity.com/paper/2007.05405/) (2020)
- [Constrained Deep Weak Supervision for Histopathology Image Segmentation](https://mlsanity.com/paper/1701.00794/) (2017)
- [DeepPhase: Surgical Phase Recognition in CATARACTS Videos](https://mlsanity.com/paper/1807.10565/) (2018)
- [Grafit: Learning fine-grained image representations with coarse labels](https://mlsanity.com/paper/2011.12982/) (2020)
- [Multi-Task Temporal Convolutional Networks for Joint Recognition of Surgical Phases and Steps in Gastric Bypass Procedures](https://mlsanity.com/paper/2102.12218/) (2021)
- [CAI4CAI: The Rise of Contextual Artificial Intelligence in Computer Assisted Interventions](https://mlsanity.com/paper/1910.09031/) (2019)
- [Using 3D Convolutional Neural Networks to Learn Spatiotemporal Features for Automatic Surgical Gesture Recognition in Video](https://mlsanity.com/paper/1907.11454/) (2019)
- [Weakly Supervised Convolutional LSTM Approach for Tool Tracking in Laparoscopic Videos](https://mlsanity.com/paper/1812.01366/) (2018)
- [Fine-grained Angular Contrastive Learning with Coarse Labels](https://mlsanity.com/paper/2012.03515/) (2020)
- [Less is More: Surgical Phase Recognition with Less Annotations through Self-Supervised Pre-training of CNN-LSTM Networks](https://mlsanity.com/paper/1805.08569/) (2018)
- [Automatic Gesture Recognition in Robot-assisted Surgery with Reinforcement Learning and Tree Search](https://mlsanity.com/paper/2002.08718/) (2020)
- [Learning from a tiny dataset of manual annotations: a teacher/student approach for surgical phase recognition](https://mlsanity.com/paper/1812.00033/) (2018)
- [Single- and Multi-Task Architectures for Surgical Workflow Challenge at M2CAI 2016](https://mlsanity.com/paper/1610.08844/) (2016)
- [Temporal coherence-based self-supervised learning for laparoscopic workflow analysis](https://mlsanity.com/paper/1806.06811/) (2018)
- [Deep Reinforcement Learning for Surgical Gesture Segmentation and Classification](https://mlsanity.com/paper/1806.08089/) (2018)
- [Unsupervised temporal context learning using convolutional neural networks for laparoscopic workflow analysis](https://mlsanity.com/paper/1702.03684/) (2017)
- [Real-time analysis of cataract surgery videos using statistical models](https://mlsanity.com/paper/1610.05465/) (2016)

## Built on by (most cited first, 5 of 5)

- [Challenges in Multi-centric Generalization: Phase and Step Recognition in Roux-en-Y Gastric Bypass Surgery](https://mlsanity.com/paper/2312.11250/) (2023)
- [Surgical Workflow Recognition and Blocking Effectiveness Detection in Laparoscopic Liver Resections with Pringle Maneuver](https://mlsanity.com/paper/2408.10538/) (2024)
- [Generalized Recognition of Basic Surgical Actions Enables Skill Assessment and Vision-Language-Model-based Surgical Planning](https://mlsanity.com/paper/2603.12787/) (2026)
- [EndoControlMag: Robust Endoscopic Vascular Motion Magnification with Periodic Reference Resetting and Hierarchical Tissue-aware Dual-Mask Control](https://mlsanity.com/paper/2507.15292/) (2025)
- [DBT-Bleed: Dual-Branch Temporal Modeling with Key-Frame Selection for Surgical Bleeding Detection](https://mlsanity.com/paper/2606.22829/) (2026)

## Abstract (arXiv)

Automatic recognition of fine-grained surgical activities, called steps, is a challenging but crucial task for intelligent intra-operative computer assistance. The development of current vision-based activity recognition methods relies heavily on a high volume of manually annotated data. This data is difficult and time-consuming to generate and requires domain-specific knowledge. In this work, we propose to use coarser and easier-to-annotate activity labels, namely phases, as weak supervision to learn step recognition with fewer step annotated videos. We introduce a step-phase dependency loss to exploit the weak supervision signal. We then employ a Single-Stage Temporal Convolutional Network (SS-TCN) with a ResNet-50 backbone, trained in an end-to-end fashion from weakly annotated videos, for temporal activity segmentation and recognition. We extensively evaluate and show the effectiveness of the proposed method on a large video dataset consisting of 40 laparoscopic gastric bypass procedures and the public benchmark CATARACTS containing 50 cataract surgeries.