# Less is More: Surgical Phase Recognition from Timestamp Supervision

arXiv: https://arxiv.org/abs/2202.08199 · HTML: https://mlsanity.com/paper/2202.08199/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 45 of its 72 references that are in the map.
- First posted on arXiv 16 Feb 2022.

## Builds on (most cited first, 30 of 45)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentation](https://mlsanity.com/paper/1903.01945/) (2019)
- [EndoNet: A Deep Architecture for Recognition Tasks on Laparoscopic Videos](https://mlsanity.com/paper/1602.03012/) (2016)
- [Aleatoric uncertainty estimation with test-time augmentation for medical image segmentation with convolutional neural networks](https://mlsanity.com/paper/1807.07356/) (2018)
- [Hide-and-Seek: Forcing a Network to be Meticulous for Weakly-supervised Object and Action Localization](https://mlsanity.com/paper/1704.04232/) (2017)
- [TeCNO: Surgical Phase Recognition with Multi-Stage Temporal Convolutional Networks](https://mlsanity.com/paper/2003.10751/) (2020)
- [Transformation Consistent Self-ensembling Model for Semi-supervised Medical Image Segmentation](https://mlsanity.com/paper/1903.00348/) (2019)
- [UntrimmedNets for Weakly Supervised Action Recognition and Detection](https://mlsanity.com/paper/1703.03329/) (2017)
- [Surgical Data Science: Enabling Next-Generation Surgery](https://mlsanity.com/paper/1701.06482/) (2017)
- [Weakly Supervised Action Localization by Sparse Temporal Pooling Network](https://mlsanity.com/paper/1712.05080/) (2017)
- [Trans-SVNet: Accurate Phase Recognition from Surgical Videos via Hybrid Embedding Aggregation Transformer](https://mlsanity.com/paper/2103.09712/) (2021)
- [W-TALC: Weakly-supervised Temporal Activity Localization and Classification](https://mlsanity.com/paper/1807.10418/) (2018)
- [Multi-Task Recurrent Convolutional Network with Correlation Loss for Surgical Video Analysis](https://mlsanity.com/paper/1907.06099/) (2019)
- [Connectionist Temporal Modeling for Weakly Supervised Action Labeling](https://mlsanity.com/paper/1607.08584/) (2016)
- [Temporal Memory Relation Network for Workflow Recognition from Surgical Video](https://mlsanity.com/paper/2103.16327/) (2021)
- [SF-Net: Single-Frame Supervision for Temporal Action Localization](https://mlsanity.com/paper/2003.06845/) (2020)
- [OperA: Attention-Regularized Transformers for Surgical Phase Recognition](https://mlsanity.com/paper/2103.03873/) (2021)
- [Weakly Supervised Energy-Based Learning for Action Segmentation](https://mlsanity.com/paper/1909.13155/) (2019)
- [Enhancing Pseudo Label Quality for Semi-Supervised Domain-Generalized Medical Image Segmentation](https://mlsanity.com/paper/2201.08657/) (2022)
- [Towards Unified Surgical Skill Assessment](https://mlsanity.com/paper/2106.01035/) (2021)
- [Weakly Supervised Action Labeling in Videos Under Ordering Constraints](https://mlsanity.com/paper/1407.1208/) (2014)
- [Temporal Action Segmentation from Timestamp Supervision](https://mlsanity.com/paper/2103.06669/) (2021)
- [Using 3D Convolutional Neural Networks to Learn Spatiotemporal Features for Automatic Surgical Gesture Recognition in Video](https://mlsanity.com/paper/1907.11454/) (2019)

## Built on by (most cited first, 13 of 13)

- [Surgical Workflow Recognition and Blocking Effectiveness Detection in Laparoscopic Liver Resections with Pringle Maneuver](https://mlsanity.com/paper/2408.10538/) (2024)
- [Free Lunch for Surgical Video Understanding by Distilling Self-Supervisions](https://mlsanity.com/paper/2205.09292/) (2022)
- [Surgical Temporal Action-aware Network with Sequence Regularization for Phase Recognition](https://mlsanity.com/paper/2311.12603/) (2023)
- [SF-TMN: SlowFast Temporal Modeling Network for Surgical Phase Recognition](https://mlsanity.com/paper/2306.08859/) (2023)
- [DACAT: Dual-stream Adaptive Clip-aware Time Modeling for Robust Online Surgical Phase Recognition](https://mlsanity.com/paper/2409.06217/) (2024)
- [Friends Across Time: Multi-Scale Action Segmentation Transformer for Surgical Phase Recognition](https://mlsanity.com/paper/2401.11644/) (2024)
- [SurgPLAN++: Universal Surgical Phase Localization Network for Online and Offline Inference](https://mlsanity.com/paper/2409.12467/) (2024)
- [Robust Surgical Phase Recognition From Annotation Efficient Supervision](https://mlsanity.com/paper/2406.18481/) (2024)
- [Data-efficient Performance Modeling via Pre-training](https://mlsanity.com/paper/2501.14438/) (2025)
- [CPKD: Clinical Prior Knowledge-Constrained Diffusion Models for Surgical Phase Recognition in Endoscopic Submucosal Dissection](https://mlsanity.com/paper/2507.03295/) (2025)
- [SurgicalMamba: Dual-Path SSD with State Regramming for Online Surgical Phase Recognition](https://mlsanity.com/paper/2605.14889/) (2026)
- [ST-ColoNet: Spatio-Temporal Colon Segment Recognition via Hybrid Attention and Edge-Guided Feature Learning](https://mlsanity.com/paper/2605.28119/) (2026)
- [Extended KAFR: A kinematic-adaptive paradigm for the efficient analysis of surgical video](https://mlsanity.com/paper/2608.01058/) (2026)

## Abstract (arXiv)

Surgical phase recognition is a fundamental task in computer-assisted surgery systems. Most existing works are under the supervision of expensive and time-consuming full annotations, which require the surgeons to repeat watching videos to find the precise start and end time for a surgical phase. In this paper, we introduce timestamp supervision for surgical phase recognition to train the models with timestamp annotations, where the surgeons are asked to identify only a single timestamp within the temporal boundary of a phase. This annotation can significantly reduce the manual annotation cost compared to the full annotations. To make full use of such timestamp supervisions, we propose a novel method called uncertainty-aware temporal diffusion (UATD) to generate trustworthy pseudo labels for training. Our proposed UATD is motivated by the property of surgical videos, i.e., the phases are long events consisting of consecutive frames. To be specific, UATD diffuses the single labelled timestamp to its corresponding high confident ( i.e., low uncertainty) neighbour frames in an iterative way. Our study uncovers unique insights of surgical phase recognition with timestamp supervisions: 1) timestamp annotation can reduce 74% annotation time compared with the full annotation, and surgeons tend to annotate those timestamps near the middle of phases; 2) extensive experiments demonstrate that our method can achieve competitive results compared with full supervision methods, while reducing manual annotation cost; 3) less is more in surgical phase recognition, i.e., less but discriminative pseudo labels outperform full but containing ambiguous frames; 4) the proposed UATD can be used as a plug and play method to clean ambiguous labels near boundaries between phases, and improve the performance of the current surgical phase recognition methods.