# Learning from a tiny dataset of manual annotations: a teacher/student approach for surgical phase recognition

arXiv: https://arxiv.org/abs/1812.00033 · HTML: https://mlsanity.com/paper/1812.00033/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Nov 2018.

## Built on by (most cited first, 21 of 21)

- [Rendezvous: Attention Mechanisms for the Recognition of Surgical Action Triplets in Endoscopic Videos](https://mlsanity.com/paper/2109.03223/) (2021)
- [CholecTriplet2021: A benchmark challenge for surgical action triplet recognition](https://mlsanity.com/paper/2204.04746/) (2022)
- [Multi-Task Temporal Convolutional Networks for Joint Recognition of Surgical Phases and Steps in Gastric Bypass Procedures](https://mlsanity.com/paper/2102.12218/) (2021)
- [OphNet: A Large-Scale Video Benchmark for Ophthalmic Surgical Workflow Understanding](https://mlsanity.com/paper/2406.07471/) (2024)
- [Gesture Recognition in Robotic Surgery: a Review](https://mlsanity.com/paper/2102.00027/) (2021)
- [Image Compositing for Segmentation of Surgical Tools without Manual Annotations](https://mlsanity.com/paper/2102.09528/) (2021)
- [CholecTriplet2022: Show me a tool and tell me the triplet -- an endoscopic vision challenge for surgical action triplet detection](https://mlsanity.com/paper/2302.06294/) (2023)
- [Federated Cycling (FedCy): Semi-supervised Federated Learning of Surgical Phases](https://mlsanity.com/paper/2203.07345/) (2022)
- [Less is More: Surgical Phase Recognition from Timestamp Supervision](https://mlsanity.com/paper/2202.08199/) (2022)
- [A Kinematic Bottleneck Approach For Pose Regression of Flexible Surgical Instruments directly from Images](https://mlsanity.com/paper/2103.00586/) (2021)
- [Self-distillation for surgical action recognition](https://mlsanity.com/paper/2303.12915/) (2023)
- [WS-SfMLearner: Self-supervised Monocular Depth and Ego-motion Estimation on Surgical Videos with Unknown Camera Parameters](https://mlsanity.com/paper/2308.11776/) (2023)
- [SurgBench: A Unified Large-Scale Benchmark for Surgical Video Analysis](https://mlsanity.com/paper/2506.07603/) (2025)
- [MedGRPO: Multi-Task Reinforcement Learning for Heterogeneous Medical Video Understanding](https://mlsanity.com/paper/2512.06581/) (2025)
- [Weakly Supervised Temporal Convolutional Networks for Fine-grained Surgical Activity Recognition](https://mlsanity.com/paper/2302.10834/) (2023)
- [Learning Invariant Representation of Tasks for Robust Surgical State Estimation](https://mlsanity.com/paper/2102.09119/) (2021)
- [A Novel Deep ML Architecture by Integrating Visual Simultaneous Localization and Mapping (vSLAM) into Mask R-CNN for Real-time Surgical Video Analysis](https://mlsanity.com/paper/2103.16847/) (2021)
- [Rethinking Generalization Performance of Surgical Phase Recognition with Expert-Generated Annotations](https://mlsanity.com/paper/2110.11626/) (2021)
- [Deep-Learning-Assisted Analysis of Cataract Surgery Videos](https://mlsanity.com/paper/2312.05900/) (2023)
- [Robust Surgical Phase Recognition From Annotation Efficient Supervision](https://mlsanity.com/paper/2406.18481/) (2024)
- [Self-supervised Learning via Cluster Distance Prediction for Operating Room Context Awareness](https://mlsanity.com/paper/2407.05448/) (2024)

## Abstract (arXiv)

Vision algorithms capable of interpreting scenes from a real-time video stream are necessary for computer-assisted surgery systems to achieve context-aware behavior. In laparoscopic procedures one particular algorithm needed for such systems is the identification of surgical phases, for which the current state of the art is a model based on a CNN-LSTM. A number of previous works using models of this kind have trained them in a fully supervised manner, requiring a fully annotated dataset. Instead, our work confronts the problem of learning surgical phase recognition in scenarios presenting scarce amounts of annotated data (under 25% of all available video recordings). We propose a teacher/student type of approach, where a strong predictor called the teacher, trained beforehand on a small dataset of ground truth-annotated videos, generates synthetic annotations for a larger dataset, which another model - the student - learns from. In our case, the teacher features a novel CNN-biLSTM-CRF architecture, designed for offline inference only. The student, on the other hand, is a CNN-LSTM capable of making real-time predictions. Results for various amounts of manually annotated videos demonstrate the superiority of the new CNN-biLSTM-CRF predictor as well as improved performance from the CNN-LSTM trained using synthetic labels generated for unannotated videos. For both offline and online surgical phase recognition with very few annotated recordings available, this new teacher/student strategy provides a valuable performance improvement by efficiently leveraging the unannotated data.