# Relevance Detection in Cataract Surgery Videos by Spatio-Temporal Action Localization

arXiv: https://arxiv.org/abs/2104.14280 · HTML: https://mlsanity.com/paper/2104.14280/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 25 references that are in the map.
- First posted on arXiv 29 Apr 2021.

## Builds on (most cited first, 9 of 9)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [EndoNet: A Deep Architecture for Recognition Tasks on Laparoscopic Videos](https://mlsanity.com/paper/1602.03012/) (2016)
- [LSTA: Long Short-Term Attention for Egocentric Action Recognition](https://mlsanity.com/paper/1811.10698/) (2018)
- [DeepPhase: Surgical Phase Recognition in CATARACTS Videos](https://mlsanity.com/paper/1807.10565/) (2018)
- [Surgical Phase Recognition of Short Video Shots Based on Temporal Modeling of Deep Features](https://mlsanity.com/paper/1807.07853/) (2018)

## Built on by (most cited first, 15 of 15)

- [OphNet: A Large-Scale Video Benchmark for Ophthalmic Surgical Workflow Understanding](https://mlsanity.com/paper/2406.07471/) (2024)
- [LensID: A CNN-RNN-Based Framework Towards Lens Irregularity Detection in Cataract Surgery Videos](https://mlsanity.com/paper/2107.00875/) (2021)
- [Domain Adaptation for Medical Image Segmentation using Transformation-Invariant Self-Training](https://mlsanity.com/paper/2307.16660/) (2023)
- [MedGen: Unlocking Medical Video Generation by Scaling Granularly-annotated Medical Videos](https://mlsanity.com/paper/2507.05675/) (2025)
- [ReCal-Net: Joint Region-Channel-Wise Calibrated Network for Semantic Segmentation in Cataract Surgery Videos](https://mlsanity.com/paper/2109.12448/) (2021)
- [Cataract-1K: Cataract Surgery Dataset for Scene Segmentation, Phase Recognition, and Irregularity Detection](https://mlsanity.com/paper/2312.06295/) (2023)
- [SegSTRONG-C: Segmenting Surgical Tools Robustly On Non-adversarial Generated Corruptions -- An EndoVis'24 Challenge](https://mlsanity.com/paper/2407.11906/) (2024)
- [DeepPyramid: Enabling Pyramid View and Deformable Pyramid Reception for Semantic Segmentation in Cataract Surgery Videos](https://mlsanity.com/paper/2207.01453/) (2022)
- [Event Recognition in Laparoscopic Gynecology Videos with Hybrid Transformers](https://mlsanity.com/paper/2312.00593/) (2023)
- [Dual Invariance Self-training for Reliable Semi-supervised Surgical Phase Recognition](https://mlsanity.com/paper/2501.17628/) (2025)
- [Artificial Intelligence for Biomedical Video Generation](https://mlsanity.com/paper/2411.07619/) (2024)
- [Predicting Postoperative Intraocular Lens Dislocation in Cataract Surgery via Deep Learning](https://mlsanity.com/paper/2312.03401/) (2023)
- [DeepPyram: Enabling Pyramid View and Deformable Pyramid Reception for Semantic Segmentation in Cataract Surgery Videos](https://mlsanity.com/paper/2109.05352/) (2021)
- [WetCat: Enabling Automated Skill Assessment in Wet-Lab Cataract Surgery Videos](https://mlsanity.com/paper/2506.08896/) (2025)
- [Data-Efficient Learning for Generalizable Surgical Video Understanding](https://mlsanity.com/paper/2508.10215/) (2025)

## Abstract (arXiv)

In cataract surgery, the operation is performed with the help of a microscope. Since the microscope enables watching real-time surgery by up to two people only, a major part of surgical training is conducted using the recorded videos. To optimize the training procedure with the video content, the surgeons require an automatic relevance detection approach. In addition to relevance-based retrieval, these results can be further used for skill assessment and irregularity detection in cataract surgery videos. In this paper, a three-module framework is proposed to detect and classify the relevant phase segments in cataract videos. Taking advantage of an idle frame recognition network, the video is divided into idle and action segments. To boost the performance in relevance detection, the cornea where the relevant surgical actions are conducted is detected in all frames using Mask R-CNN. The spatiotemporally localized segments containing higher-resolution information about the pupil texture and actions, and complementary temporal information from the same phase are fed into the relevance detection module. This module consists of four parallel recurrent CNNs being responsible to detect four relevant phases that have been defined with medical experts. The results will then be integrated to classify the action phases as irrelevant or one of four relevant phases. Experimental results reveal that the proposed approach outperforms static CNNs and different configurations of feature-based and end-to-end recurrent networks.