# An Empirical Study on Activity Recognition in Long Surgical Videos

arXiv: https://arxiv.org/abs/2205.02805 · HTML: https://mlsanity.com/paper/2205.02805/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 32 references that are in the map.
- First posted on arXiv 5 May 2022.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [On the Properties of Neural Machine Translation: Encoder-Decoder Approaches](https://mlsanity.com/paper/1409.1259/) (2014)
- [Video Swin Transformer](https://mlsanity.com/paper/2106.13230/) (2021)
- [Temporal Convolutional Networks for Action Segmentation and Detection](https://mlsanity.com/paper/1611.05267/) (2016)
- [EndoNet: A Deep Architecture for Recognition Tasks on Laparoscopic Videos](https://mlsanity.com/paper/1602.03012/) (2016)
- [TeCNO: Surgical Phase Recognition with Multi-Stage Temporal Convolutional Networks](https://mlsanity.com/paper/2003.10751/) (2020)
- [The Best of Both Worlds: Combining Recent Advances in Neural Machine Translation](https://mlsanity.com/paper/1804.09849/) (2018)
- [Multi-Task Recurrent Convolutional Network with Correlation Loss for Surgical Video Analysis](https://mlsanity.com/paper/1907.06099/) (2019)
- [Automatic Operating Room Surgical Activity Recognition for Robot-Assisted Surgery](https://mlsanity.com/paper/2006.16166/) (2020)
- [Less is More: Surgical Phase Recognition with Less Annotations through Self-Supervised Pre-training of CNN-LSTM Networks](https://mlsanity.com/paper/1805.08569/) (2018)

## Built on by (most cited first, 13 of 13)

- [LVAgent: Long Video Understanding by Multi-Round Dynamical Collaboration of MLLM Agents](https://mlsanity.com/paper/2503.10200/) (2025)
- [Metrics Matter in Surgical Phase Recognition](https://mlsanity.com/paper/2305.13961/) (2023)
- [SurgMAE: Masked Autoencoders for Long Surgical Video Analysis](https://mlsanity.com/paper/2305.11451/) (2023)
- [TUNeS: A Temporal U-Net with Self-Attention for Video-based Surgical Phase Recognition](https://mlsanity.com/paper/2307.09997/) (2023)
- [ST(OR)2: Spatio-Temporal Object Level Reasoning for Activity Recognition in the Operating Room](https://mlsanity.com/paper/2312.12250/) (2023)
- [VideoChat-M1: Collaborative Policy Planning for Video Understanding via Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2511.19524/) (2025)
- [SF-TMN: SlowFast Temporal Modeling Network for Surgical Phase Recognition](https://mlsanity.com/paper/2306.08859/) (2023)
- [Event Recognition in Laparoscopic Gynecology Videos with Hybrid Transformers](https://mlsanity.com/paper/2312.00593/) (2023)
- [Self-Knowledge Distillation for Surgical Phase Recognition](https://mlsanity.com/paper/2306.08961/) (2023)
- [M33D: Learning 3D priors using Multi-Modal Masked Autoencoders for 2D image and video understanding](https://mlsanity.com/paper/2309.15313/) (2023)
- [Friends Across Time: Multi-Scale Action Segmentation Transformer for Surgical Phase Recognition](https://mlsanity.com/paper/2401.11644/) (2024)
- [SurgicalMamba: Dual-Path SSD with State Regramming for Online Surgical Phase Recognition](https://mlsanity.com/paper/2605.14889/) (2026)
- [Extended KAFR: A kinematic-adaptive paradigm for the efficient analysis of surgical video](https://mlsanity.com/paper/2608.01058/) (2026)

## Abstract (arXiv)

Activity recognition in surgical videos is a key research area for developing next-generation devices and workflow monitoring systems. Since surgeries are long processes with highly-variable lengths, deep learning models used for surgical videos often consist of a two-stage setup using a backbone and temporal sequence model. In this paper, we investigate many state-of-the-art backbones and temporal models to find architectures that yield the strongest performance for surgical activity recognition. We first benchmark the models performance on a large-scale activity recognition dataset containing over 800 surgery videos captured in multiple clinical operating rooms. We further evaluate the models on the two smaller public datasets, the Cholec80 and Cataract-101 datasets, containing only 80 and 101 videos respectively. We empirically found that Swin-Transformer+BiGRU temporal model yielded strong performance on both datasets. Finally, we investigate the adaptability of the model to new domains by fine-tuning models to a new hospital and experimenting with a recent unsupervised domain adaptation approach.