# PitVis-2023 Challenge: Workflow Recognition in videos of Endoscopic Pituitary Surgery arXiv: https://arxiv.org/abs/2409.01184 ยท HTML: https://mlsanity.com/paper/2409.01184/ ## Key facts - Cited by 10 papers in the map (data through 14 Oct 2026). - Builds on 21 of its 44 references that are in the map. - First posted on arXiv 2 Sep 2024. ## Builds on (most cited first, 21 of 21) - [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015) - [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021) - [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016) - [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019) - [YOLOv4: Optimal Speed and Accuracy of Object Detection](https://mlsanity.com/paper/2004.10934/) (2020) - [DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection](https://mlsanity.com/paper/2203.03605/) (2022) - [Multiscale Vision Transformers](https://mlsanity.com/paper/2104.11227/) (2021) - [EVA-02: A Visual Representation for Neon Genesis](https://mlsanity.com/paper/2303.11331/) (2023) - [EndoNet: A Deep Architecture for Recognition Tasks on Laparoscopic Videos](https://mlsanity.com/paper/1602.03012/) (2016) - [XCiT: Cross-Covariance Image Transformers](https://mlsanity.com/paper/2106.09681/) (2021) - [TinyViT: Fast Pretraining Distillation for Small Vision Transformers](https://mlsanity.com/paper/2207.10666/) (2022) - [TeCNO: Surgical Phase Recognition with Multi-Stage Temporal Convolutional Networks](https://mlsanity.com/paper/2003.10751/) (2020) - [Surgical Data Science -- from Concepts toward Clinical Translation](https://mlsanity.com/paper/2011.02284/) (2020) - [Segmental Spatiotemporal CNNs for Fine-grained Action Segmentation](https://mlsanity.com/paper/1602.02995/) (2016) - [Multi-Task Recurrent Convolutional Network with Correlation Loss for Surgical Video Analysis](https://mlsanity.com/paper/1907.06099/) (2019) - [BIAS: Transparent reporting of biomedical image analysis challenges](https://mlsanity.com/paper/1910.04071/) (2019) - [PitVQA: Image-grounded Text Embedding LLM for Visual Question Answering in Pituitary Surgery](https://mlsanity.com/paper/2405.13949/) (2024) - [Methods and datasets for segmentation of minimally invasive surgical instruments in endoscopic images and videos: A review of the state of the art](https://mlsanity.com/paper/2304.13014/) (2023) - [Aggregating Long-Term Context for Learning Laparoscopic and Robot-Assisted Surgical Workflows](https://mlsanity.com/paper/2009.00681/) (2020) - [ARST: Auto-Regressive Surgical Transformer for Phase Recognition from Laparoscopic Videos](https://mlsanity.com/paper/2209.01148/) (2022) - [Multitask Learning in Minimally Invasive Surgical Vision: A Review](https://mlsanity.com/paper/2401.08256/) (2024) ## Built on by (most cited first, 10 of 10) - [How Far Are Surgeons from Surgical World Models? A Pilot Study on Zero-shot Surgical Video Generation with Expert Assessment](https://mlsanity.com/paper/2511.01775/) (2025) - [Large-scale Self-supervised Video Foundation Model for Intelligent Surgery](https://mlsanity.com/paper/2506.02692/) (2025) - [MedGen: Unlocking Medical Video Generation by Scaling Granularly-annotated Medical Videos](https://mlsanity.com/paper/2507.05675/) (2025) - [SurgMotion: A Video-Native Foundation Model for Universal Understanding of Surgical Videos](https://mlsanity.com/paper/2602.05638/) (2026) - [Conformal forecasting for surgical instrument trajectory](https://mlsanity.com/paper/2503.04191/) (2025) - [PitRSDNet: Predicting Intra-operative Remaining Surgery Duration in Endoscopic Pituitary Surgery](https://mlsanity.com/paper/2409.16998/) (2024) - [Multi-Modal Explainable Medical AI Assistant for Trustworthy Human-AI Collaboration](https://mlsanity.com/paper/2505.06898/) (2025) - [Adenocarcinoma Segmentation Using Pre-trained Swin-UNet with Parallel Cross-Attention for Multi-Domain Imaging](https://mlsanity.com/paper/2409.15501/) (2024) - [Automated Surgical Skill Assessment in Endoscopic Pituitary Surgery using Real-time Instrument Tracking on a High-fidelity Bench-top Phantom](https://mlsanity.com/paper/2409.17025/) (2024) - [Strategy-Supervised Autonomous Laparoscopic Camera Control via Event-Driven Graph Mining](https://mlsanity.com/paper/2602.20500/) (2026) ## Abstract (arXiv) The field of computer vision applied to videos of minimally invasive surgery is ever-growing. Workflow recognition pertains to the automated recognition of various aspects of a surgery: including which surgical steps are performed; and which surgical instruments are used. This information can later be used to assist clinicians when learning the surgery; during live surgery; and when writing operation notes. The Pituitary Vision (PitVis) 2023 Challenge tasks the community to step and instrument recognition in videos of endoscopic pituitary surgery. This is a unique task when compared to other minimally invasive surgeries due to the smaller working space, which limits and distorts vision; and higher frequency of instrument and step switching, which requires more precise model predictions. Participants were provided with 25-videos, with results presented at the MICCAI-2023 conference as part of the Endoscopic Vision 2023 Challenge in Vancouver, Canada, on 08-Oct-2023. There were 18-submissions from 9-teams across 6-countries, using a variety of deep learning models. A commonality between the top performing models was incorporating spatio-temporal and multi-task methods, with greater than 50% and 10% macro-F1-score improvement over purely spacial single-task models in step and instrument recognition respectively. The PitVis-2023 Challenge therefore demonstrates state-of-the-art computer vision models in minimally invasive surgery are transferable to a new dataset, with surgery specific techniques used to enhance performance, progressing the field further. Benchmark results are provided in the paper, and the dataset is publicly available at: https://doi.org/10.5522/04/26531686.