# Trans-SVNet: Accurate Phase Recognition from Surgical Videos via Hybrid Embedding Aggregation Transformer

arXiv: https://arxiv.org/abs/2103.09712 · HTML: https://mlsanity.com/paper/2103.09712/

## Key facts

- Cited by 94 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 33 references that are in the map.
- First posted on arXiv 17 Mar 2021.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021)
- [Temporal Convolutional Networks for Action Segmentation and Detection](https://mlsanity.com/paper/1611.05267/) (2016)
- [MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentation](https://mlsanity.com/paper/1903.01945/) (2019)
- [EndoNet: A Deep Architecture for Recognition Tasks on Laparoscopic Videos](https://mlsanity.com/paper/1602.03012/) (2016)
- [Deep Closest Point: Learning Representations for Point Cloud Registration](https://mlsanity.com/paper/1905.03304/) (2019)
- [Temporal Convolutional Networks: A Unified Approach to Action Segmentation](https://mlsanity.com/paper/1608.08242/) (2016)
- [TeCNO: Surgical Phase Recognition with Multi-Stage Temporal Convolutional Networks](https://mlsanity.com/paper/2003.10751/) (2020)
- [Surgical Data Science: Enabling Next-Generation Surgery](https://mlsanity.com/paper/1701.06482/) (2017)
- [Multi-Task Recurrent Convolutional Network with Correlation Loss for Surgical Video Analysis](https://mlsanity.com/paper/1907.06099/) (2019)
- [Incorporating Temporal Prior from Motion Flow for Instrument Segmentation in Minimally Invasive Surgery Video](https://mlsanity.com/paper/1907.07899/) (2019)
- [DeepPhase: Surgical Phase Recognition in CATARACTS Videos](https://mlsanity.com/paper/1807.10565/) (2018)
- [Using 3D Convolutional Neural Networks to Learn Spatiotemporal Features for Automatic Surgical Gesture Recognition in Video](https://mlsanity.com/paper/1907.11454/) (2019)
- [Learning Motion Flows for Semi-supervised Instrument Segmentation from Robotic Surgical Video](https://mlsanity.com/paper/2007.02501/) (2020)
- [Automatic Gesture Recognition in Robot-assisted Surgery with Reinforcement Learning and Tree Search](https://mlsanity.com/paper/2002.08718/) (2020)
- [Single- and Multi-Task Architectures for Surgical Workflow Challenge at M2CAI 2016](https://mlsanity.com/paper/1610.08844/) (2016)
- [Symmetric Dilated Convolution for Surgical Gesture Recognition](https://mlsanity.com/paper/2007.06373/) (2020)
- [Real-time analysis of cataract surgery videos using statistical models](https://mlsanity.com/paper/1610.05465/) (2016)

## Built on by (most cited first, 30 of 94)

- [Transformers in Medical Imaging: A Survey](https://mlsanity.com/paper/2201.09873/) (2022)
- [Rendezvous: Attention Mechanisms for the Recognition of Surgical Action Triplets in Endoscopic Videos](https://mlsanity.com/paper/2109.03223/) (2021)
- [AutoLaparo: A New Dataset of Integrated Multi-tasks for Image-guided Surgical Automation in Laparoscopic Hysterectomy](https://mlsanity.com/paper/2208.02049/) (2022)
- [LoViT: Long Video Transformer for Surgical Phase Recognition](https://mlsanity.com/paper/2305.08989/) (2023)
- [SAR-RARP50: Segmentation of surgical instrumentation and Action Recognition on Robot-Assisted Radical Prostatectomy Challenge](https://mlsanity.com/paper/2401.00496/) (2023)
- [Pixel-Wise Recognition for Holistic Surgical Scene Understanding](https://mlsanity.com/paper/2401.11174/) (2024)
- [TraSeTR: Track-to-Segment Transformer with Contrastive Query for Instance-level Instrument Segmentation in Robotic Surgery](https://mlsanity.com/paper/2202.08453/) (2022)
- [Procedure-Aware Surgical Video-language Pretraining with Hierarchical Knowledge Augmentation](https://mlsanity.com/paper/2410.00263/) (2024)
- [Towards Holistic Surgical Scene Understanding](https://mlsanity.com/paper/2212.04582/) (2022)
- [Surgformer: Surgical Transformer with Hierarchical Temporal Attention for Surgical Phase Recognition](https://mlsanity.com/paper/2408.03867/) (2024)
- [Exploring Intra- and Inter-Video Relation for Surgical Semantic Scene Segmentation](https://mlsanity.com/paper/2203.15251/) (2022)
- [Rendezvous in Time: An Attention-based Temporal Fusion approach for Surgical Triplet Recognition](https://mlsanity.com/paper/2211.16963/) (2022)
- [Exploring Segment-level Semantics for Online Phase Recognition from Surgical Videos](https://mlsanity.com/paper/2111.11044/) (2021)
- [Data Splits and Metrics for Method Benchmarking on Surgical Action Triplet Datasets](https://mlsanity.com/paper/2204.05235/) (2022)
- [CholecTriplet2022: Show me a tool and tell me the triplet -- an endoscopic vision challenge for surgical action triplet detection](https://mlsanity.com/paper/2302.06294/) (2023)
- [LABRAD-OR: Lightweight Memory Scene Graphs for Accurate Bimodal Reasoning in Dynamic Operating Rooms](https://mlsanity.com/paper/2303.13293/) (2023)
- [Dynamic Scene Graph Representation for Surgical Video](https://mlsanity.com/paper/2309.14538/) (2023)
- [Not End-to-End: Explore Multi-Stage Architecture for Online Surgical Phase Recognition](https://mlsanity.com/paper/2107.04810/) (2021)
- [EgoSurgery-Phase: A Dataset of Surgical Phase Recognition from Egocentric Open Surgery Videos](https://mlsanity.com/paper/2405.19644/) (2024)
- [On the Pitfalls of Batch Normalization for End-to-End Video Learning: A Study on Surgical Workflow Analysis](https://mlsanity.com/paper/2203.07976/) (2022)
- [Federated Cycling (FedCy): Semi-supervised Federated Learning of Surgical Phases](https://mlsanity.com/paper/2203.07345/) (2022)
- [SurgLaVi: Large-Scale Hierarchical Dataset for Surgical Vision-Language Representation Learning](https://mlsanity.com/paper/2509.10555/) (2025)
- [Metrics Matter in Surgical Phase Recognition](https://mlsanity.com/paper/2305.13961/) (2023)
- [Encoding Surgical Videos as Latent Spatiotemporal Graphs for Object and Anatomy-Driven Reasoning](https://mlsanity.com/paper/2312.06829/) (2023)
- [Less is More: Surgical Phase Recognition from Timestamp Supervision](https://mlsanity.com/paper/2202.08199/) (2022)
- [Recognition and Prediction of Surgical Gestures and Trajectories Using Transformer Models in Robot-Assisted Surgery](https://mlsanity.com/paper/2212.01683/) (2022)
- [SurgPLAN: Surgical Phase Localization Network for Phase Recognition](https://mlsanity.com/paper/2311.09965/) (2023)
- [VidLPRO: A underlineVideo-underlineLanguage underlinePre-training Framework for underlineRobotic and Laparoscopic Surgery](https://mlsanity.com/paper/2409.04732/) (2024)
- [EndoMamba: An Efficient Foundation Model for Endoscopic Videos via Hierarchical Pre-training](https://mlsanity.com/paper/2502.19090/) (2025)
- [Self-distillation for surgical action recognition](https://mlsanity.com/paper/2303.12915/) (2023)

## Abstract (arXiv)

Real-time surgical phase recognition is a fundamental task in modern operating rooms. Previous works tackle this task relying on architectures arranged in spatio-temporal order, however, the supportive benefits of intermediate spatial features are not considered. In this paper, we introduce, for the first time in surgical workflow analysis, Transformer to reconsider the ignored complementary effects of spatial and temporal features for accurate surgical phase recognition. Our hybrid embedding aggregation Transformer fuses cleverly designed spatial and temporal embeddings by allowing for active queries based on spatial information from temporal embedding sequences. More importantly, our framework processes the hybrid embeddings in parallel to achieve a high inference speed. Our method is thoroughly validated on two large surgical video datasets, i.e., Cholec80 and M2CAI16 Challenge datasets, and outperforms the state-of-the-art approaches at a processing speed of 91 fps.