# EndoNet: A Deep Architecture for Recognition Tasks on Laparoscopic Videos

arXiv: https://arxiv.org/abs/1602.03012 · HTML: https://mlsanity.com/paper/1602.03012/

## Key facts

- Cited by 367 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Feb 2016.

## Built on by (most cited first, 30 of 367)

- [Rendezvous: Attention Mechanisms for the Recognition of Surgical Action Triplets in Endoscopic Videos](https://mlsanity.com/paper/2109.03223/) (2021)
- [Trans-SVNet: Accurate Phase Recognition from Surgical Videos via Hybrid Embedding Aggregation Transformer](https://mlsanity.com/paper/2103.09712/) (2021)
- [AutoLaparo: A New Dataset of Integrated Multi-tasks for Image-guided Surgical Automation in Laparoscopic Hysterectomy](https://mlsanity.com/paper/2208.02049/) (2022)
- [Learning Multi-modal Representations by Watching Hundreds of Surgical Video Lectures](https://mlsanity.com/paper/2307.15220/) (2023)
- [Temporal Memory Relation Network for Workflow Recognition from Surgical Video](https://mlsanity.com/paper/2103.16327/) (2021)
- [OperA: Attention-Regularized Transformers for Surgical Phase Recognition](https://mlsanity.com/paper/2103.03873/) (2021)
- [Comparative Validation of Machine Learning Algorithms for Surgical Workflow and Skill Analysis with the HeiChole Benchmark](https://mlsanity.com/paper/2109.14956/) (2021)
- [Surgical-VQA: Visual Question Answering in Surgical Scenes using Transformer](https://mlsanity.com/paper/2206.11053/) (2022)
- [LoViT: Long Video Transformer for Surgical Phase Recognition](https://mlsanity.com/paper/2305.08989/) (2023)
- [Pixel-Wise Recognition for Holistic Surgical Scene Understanding](https://mlsanity.com/paper/2401.11174/) (2024)
- [Dissecting Self-Supervised Learning Methods for Surgical Computer Vision](https://mlsanity.com/paper/2207.00449/) (2022)
- [SurgicalGPT: End-to-End Language-Vision GPT for Visual Question Answering in Surgery](https://mlsanity.com/paper/2304.09974/) (2023)
- [CholecTriplet2021: A benchmark challenge for surgical action triplet recognition](https://mlsanity.com/paper/2204.04746/) (2022)
- [Multi-Task Temporal Convolutional Networks for Joint Recognition of Surgical Phases and Steps in Gastric Bypass Procedures](https://mlsanity.com/paper/2102.12218/) (2021)
- [HecVL: Hierarchical Video-Language Pretraining for Zero-shot Surgical Phase Recognition](https://mlsanity.com/paper/2405.10075/) (2024)
- [Advancing Surgical VQA with Scene Graph Knowledge](https://mlsanity.com/paper/2312.10251/) (2023)
- [Challenges in Multi-centric Generalization: Phase and Step Recognition in Roux-en-Y Gastric Bypass Surgery](https://mlsanity.com/paper/2312.11250/) (2023)
- [Procedure-Aware Surgical Video-language Pretraining with Hierarchical Knowledge Augmentation](https://mlsanity.com/paper/2410.00263/) (2024)
- [The SARAS Endoscopic Surgeon Action Detection (ESAD) dataset: Challenges and methods](https://mlsanity.com/paper/2104.03178/) (2021)
- [EndoChat: Grounded Multimodal Large Language Model for Endoscopic Surgery](https://mlsanity.com/paper/2501.11347/) (2025)
- [LLaVA-Surg: Towards Multimodal Surgical Assistant via Structured Surgical Video Learning](https://mlsanity.com/paper/2408.07981/) (2024)
- [Towards Holistic Surgical Scene Understanding](https://mlsanity.com/paper/2212.04582/) (2022)
- [OphNet: A Large-Scale Video Benchmark for Ophthalmic Surgical Workflow Understanding](https://mlsanity.com/paper/2406.07471/) (2024)
- [Surgformer: Surgical Transformer with Hierarchical Temporal Attention for Surgical Phase Recognition](https://mlsanity.com/paper/2408.03867/) (2024)
- [SurgVLM: A Large Vision-Language Model and Systematic Evaluation Benchmark for Surgical Intelligence](https://mlsanity.com/paper/2506.02555/) (2025)
- [MIcro-Surgical Anastomose Workflow recognition challenge report](https://mlsanity.com/paper/2103.13111/) (2021)
- [MM-OR: A Large Multimodal Operating Room Dataset for Semantic Understanding of High-Intensity Surgical Environments](https://mlsanity.com/paper/2503.02579/) (2025)
- [Rendezvous in Time: An Attention-based Temporal Fusion approach for Surgical Triplet Recognition](https://mlsanity.com/paper/2211.16963/) (2022)
- [General surgery vision transformer: A video pre-trained foundation model for general surgery](https://mlsanity.com/paper/2403.05949/) (2024)
- [Scaling up self-supervised learning for improved surgical foundation models](https://mlsanity.com/paper/2501.09436/) (2025)

## Abstract (arXiv)

Surgical workflow recognition has numerous potential medical applications, such as the automatic indexing of surgical video databases and the optimization of real-time operating room scheduling, among others. As a result, phase recognition has been studied in the context of several kinds of surgeries, such as cataract, neurological, and laparoscopic surgeries. In the literature, two types of features are typically used to perform this task: visual features and tool usage signals. However, the visual features used are mostly handcrafted. Furthermore, the tool usage signals are usually collected via a manual annotation process or by using additional equipment. In this paper, we propose a novel method for phase recognition that uses a convolutional neural network (CNN) to automatically learn features from cholecystectomy videos and that relies uniquely on visual information. In previous studies, it has been shown that the tool signals can provide valuable information in performing the phase recognition task. Thus, we present a novel CNN architecture, called EndoNet, that is designed to carry out the phase recognition and tool presence detection tasks in a multi-task manner. To the best of our knowledge, this is the first work proposing to use a CNN for multiple recognition tasks on laparoscopic videos. Extensive experimental comparisons to other methods show that EndoNet yields state-of-the-art results for both tasks.