# Video Dataset for Surgical Phase, Keypoint, and Instrument Recognition in Laparoscopic Surgery (PhaKIR)

arXiv: https://arxiv.org/abs/2511.06549 · HTML: https://mlsanity.com/paper/2511.06549/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 26 references that are in the map.
- First posted on arXiv 9 Nov 2025.

## Builds on (most cited first, 11 of 11)

- [EndoNet: A Deep Architecture for Recognition Tasks on Laparoscopic Videos](https://mlsanity.com/paper/1602.03012/) (2016)
- [2017 Robotic Instrument Segmentation Challenge](https://mlsanity.com/paper/1902.06426/) (2019)
- [Surgical Data Science -- from Concepts toward Clinical Translation](https://mlsanity.com/paper/2011.02284/) (2020)
- [Comparative Validation of Machine Learning Algorithms for Surgical Workflow and Skill Analysis with the HeiChole Benchmark](https://mlsanity.com/paper/2109.14956/) (2021)
- [SAR-RARP50: Segmentation of surgical instrumentation and Action Recognition on Robot-Assisted Radical Prostatectomy Challenge](https://mlsanity.com/paper/2401.00496/) (2023)
- [The TUM LapChole dataset for the M2CAI 2016 workflow challenge](https://mlsanity.com/paper/1610.09278/) (2016)
- [Comparative evaluation of instrument segmentation and tracking methods in minimally invasive surgery](https://mlsanity.com/paper/1805.02475/) (2018)
- [CholecTriplet2022: Show me a tool and tell me the triplet -- an endoscopic vision challenge for surgical action triplet detection](https://mlsanity.com/paper/2302.06294/) (2023)
- [Methods and datasets for segmentation of minimally invasive surgical instruments in endoscopic images and videos: A review of the state of the art](https://mlsanity.com/paper/2304.13014/) (2023)
- [Comparative validation of surgical phase recognition, instrument keypoint estimation, and instrument instance segmentation in endoscopy: Results of the PhaKIR 2024 challenge](https://mlsanity.com/paper/2507.16559/) (2025)
- [Objective Surgical Skills Assessment and Tool Localization: Results from the MICCAI 2021 SimSurgSkill Challenge](https://mlsanity.com/paper/2212.04448/) (2022)

## Built on by (most cited first, 9 of 9)

- [SUREON: A Benchmark and Vision-Language-Model for Surgical Reasoning](https://mlsanity.com/paper/2603.06570/) (2026)
- [SurgAtlas: A Large-Scale Surgical Video-Language Dataset with 2,391 Hours of Open and Minimally Invasive Surgery](https://mlsanity.com/paper/2606.25905/) (2026)
- [On the Role of Depth in Surgical Vision Foundation Models: An Empirical Study of RGB-D Pre-training](https://mlsanity.com/paper/2601.18929/) (2026)
- [OR-Action: Multi-Role Video Understanding with Fine-Grained Actions](https://mlsanity.com/paper/2606.13332/) (2026)
- [Stitch-Inferencer: Enhance Endoscopic Video Segmentation and Tracking via Panoramic Reconstruction](https://mlsanity.com/paper/2607.14968/) (2026)
- [DART: Depth-as-Target Pretraining for Surgical Vision Foundation Models](https://mlsanity.com/paper/2609.04555/) (2026)
- [SIRA: Reasoning-Aware Surgical Instrument Segmentation via Query-Anchored Alignment](https://mlsanity.com/paper/2609.21402/) (2026)
- [LD-RSVIS: A Large-Scale and Diverse Benchmark for Referring Surgical Video Instrument Segmentation](https://mlsanity.com/paper/2609.23067/) (2026)
- [Spatiotemporal Recognition of Abdominal Wall Layers During Optical Trocar Entry](https://mlsanity.com/paper/2610.14512/) (2026)

## Abstract (arXiv)

Robotic- and computer-assisted minimally invasive surgery (RAMIS) is increasingly relying on computer vision methods for reliable instrument recognition and surgical workflow understanding. Developing such systems often requires large, well-annotated datasets, but existing resources often address isolated tasks, neglect temporal dependencies, or lack multi-center variability. We present the Surgical Procedure Phase, Keypoint, and Instrument Recognition (PhaKIR) dataset, comprising eight complete laparoscopic cholecystectomy videos recorded at three medical centers. The dataset provides frame-level annotations for three interconnected tasks: surgical phase recognition (485,875 frames), instrument keypoint estimation (19,435 frames), and instrument instance segmentation (19,435 frames). PhaKIR is, to our knowledge, the first multi-institutional dataset to jointly provide phase labels, instrument pose information, and pixel-accurate instrument segmentations, while also enabling the exploitation of temporal context since full surgical procedure sequences are available. It served as the basis for the PhaKIR Challenge as part of the Endoscopic Vision (EndoVis) Challenge at MICCAI 2024 to benchmark methods in surgical scene understanding, thereby further validating the dataset's quality and relevance. The dataset is publicly available upon request via the Zenodo platform.