# ViTALS: Vision Transformer for Action Localization in Surgical Nephrectomy

arXiv: https://arxiv.org/abs/2405.02571 · HTML: https://mlsanity.com/paper/2405.02571/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 29 references that are in the map.
- First posted on arXiv 4 May 2024.

## Builds on (most cited first, 17 of 17)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentation](https://mlsanity.com/paper/1903.01945/) (2019)
- [EndoNet: A Deep Architecture for Recognition Tasks on Laparoscopic Videos](https://mlsanity.com/paper/1602.03012/) (2016)
- [ActionFormer: Localizing Moments of Actions with Transformers](https://mlsanity.com/paper/2202.07925/) (2022)
- [Anticipative Video Transformer](https://mlsanity.com/paper/2106.02036/) (2021)
- [ASFormer: Transformer for Action Segmentation](https://mlsanity.com/paper/2110.08568/) (2021)
- [TeCNO: Surgical Phase Recognition with Multi-Stage Temporal Convolutional Networks](https://mlsanity.com/paper/2003.10751/) (2020)
- [Trans-SVNet: Accurate Phase Recognition from Surgical Videos via Hybrid Embedding Aggregation Transformer](https://mlsanity.com/paper/2103.09712/) (2021)
- [Multi-Task Recurrent Convolutional Network with Correlation Loss for Surgical Video Analysis](https://mlsanity.com/paper/1907.06099/) (2019)
- [Weakly-Supervised Action Segmentation with Iterative Soft Boundary Assignment](https://mlsanity.com/paper/1803.10699/) (2018)
- [An end-to-end generative framework for video segmentation and recognition](https://mlsanity.com/paper/1509.01947/) (2015)
- [OperA: Attention-Regularized Transformers for Surgical Phase Recognition](https://mlsanity.com/paper/2103.03873/) (2021)
- [Not End-to-End: Explore Multi-Stage Architecture for Online Surgical Phase Recognition](https://mlsanity.com/paper/2107.04810/) (2021)
- [Single- and Multi-Task Architectures for Surgical Workflow Challenge at M2CAI 2016](https://mlsanity.com/paper/1610.08844/) (2016)

## Built on by (most cited first, 2 of 2)

- [When do they StOP?: A First Step Towards Automatically Identifying Team Communication in the Operating Room](https://mlsanity.com/paper/2502.08299/) (2025)
- [Multi-scale Temporal Prediction via Incremental Generation and Multi-agent Collaboration](https://mlsanity.com/paper/2509.17429/) (2025)

## Abstract (arXiv)

Surgical action localization is a challenging computer vision problem. While it has promising applications including automated training of surgery procedures, surgical workflow optimization, etc., appropriate model design is pivotal to accomplishing this task. Moreover, the lack of suitable medical datasets adds an additional layer of complexity. To that effect, we introduce a new complex dataset of nephrectomy surgeries called UroSlice. To perform the action localization from these videos, we propose a novel model termed as `ViTALS' (Vision Transformer for Action Localization in Surgical Nephrectomy). Our model incorporates hierarchical dilated temporal convolution layers and inter-layer residual connections to capture the temporal correlations at finer as well as coarser granularities. The proposed approach achieves state-of-the-art performance on Cholec80 and UroSlice datasets (89.8% and 66.1% accuracy, respectively), validating its effectiveness.