# CholecInstanceSeg: A Tool Instance Segmentation Dataset for Laparoscopic Surgery arXiv: https://arxiv.org/abs/2406.16039 ยท HTML: https://mlsanity.com/paper/2406.16039/ ## Key facts - Cited by 16 papers in the map (data through 14 Oct 2026). - Builds on 19 of its 34 references that are in the map. - First posted on arXiv 23 Jun 2024. ## Builds on (most cited first, 19 of 19) - [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014) - [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023) - [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017) - [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021) - [Panoptic Segmentation](https://mlsanity.com/paper/1801.00868/) (2018) - [EndoNet: A Deep Architecture for Recognition Tasks on Laparoscopic Videos](https://mlsanity.com/paper/1602.03012/) (2016) - [RTMDet: An Empirical Study of Designing Real-Time Object Detectors](https://mlsanity.com/paper/2212.07784/) (2022) - [2017 Robotic Instrument Segmentation Challenge](https://mlsanity.com/paper/1902.06426/) (2019) - [Rendezvous: Attention Mechanisms for the Recognition of Surgical Action Triplets in Endoscopic Videos](https://mlsanity.com/paper/2109.03223/) (2021) - [CholecSeg8k: A Semantic Segmentation Dataset for Laparoscopic Cholecystectomy Based on Cholec80](https://mlsanity.com/paper/2012.12453/) (2020) - [AutoLaparo: A New Dataset of Integrated Multi-tasks for Image-guided Surgical Automation in Laparoscopic Hysterectomy](https://mlsanity.com/paper/2208.02049/) (2022) - [SAR-RARP50: Segmentation of surgical instrumentation and Action Recognition on Robot-Assisted Radical Prostatectomy Challenge](https://mlsanity.com/paper/2401.00496/) (2023) - [CAI4CAI: The Rise of Contextual Artificial Intelligence in Computer Assisted Interventions](https://mlsanity.com/paper/1910.09031/) (2019) - [Comparative evaluation of instrument segmentation and tracking methods in minimally invasive surgery](https://mlsanity.com/paper/1805.02475/) (2018) - [From Forks to Forceps: A New Framework for Instance Segmentation of Surgical Instruments](https://mlsanity.com/paper/2211.16200/) (2022) - [Text Promptable Surgical Instrument Segmentation with Vision-Language Models](https://mlsanity.com/paper/2306.09244/) (2023) - [RSDNet: Learning to Predict Remaining Surgery Duration from Laparoscopic Videos Without Manual Annotations](https://mlsanity.com/paper/1802.03243/) (2018) - [Real-Time Instrument Segmentation in Robotic Surgery using Auxiliary Supervised Deep Adversarial Learning](https://mlsanity.com/paper/2007.11319/) (2020) - [Robust Medical Instrument Segmentation Challenge 2019](https://mlsanity.com/paper/2003.10299/) (2020) ## Built on by (most cited first, 16 of 16) - [Comparative validation of surgical phase recognition, instrument keypoint estimation, and instrument instance segmentation in endoscopy: Results of the PhaKIR 2024 challenge](https://mlsanity.com/paper/2507.16559/) (2025) - [SUREON: A Benchmark and Vision-Language-Model for Surgical Reasoning](https://mlsanity.com/paper/2603.06570/) (2026) - [ROBUST-MIPS: A Combined Skeletal Pose and Instance Segmentation Dataset for Laparoscopic Surgical Instruments](https://mlsanity.com/paper/2508.21096/) (2025) - [Surg-R1: A Hierarchical Reasoning Foundation Model for Scalable and Interpretable Surgical Decision Support with Multi-Center Clinical Validation](https://mlsanity.com/paper/2603.12430/) (2026) - [SurgSigma: A Spectrum of Large-Scale Multimodal Data and Foundation Models for Surgical Intelligence](https://mlsanity.com/paper/2603.16822/) (2026) - [Revisiting the Evaluation Bias Introduced by Frame Sampling Strategies in Surgical Video Segmentation Using SAM2](https://mlsanity.com/paper/2502.20934/) (2025) - [Grounding Surgical Action Triplets with Instrument Instance Segmentation: A Dataset and Target-Aware Fusion Approach](https://mlsanity.com/paper/2511.00643/) (2025) - [SurgSLOT: Segment Anything in Surgical Videos via Semantic Long-term Tracking](https://mlsanity.com/paper/2511.16618/) (2025) - [SurgPIS: Surgical-instrument-level Instances and Part-level Semantics for Weakly-supervised Part-aware Instance Segmentation](https://mlsanity.com/paper/2507.19592/) (2025) - [On the Role of Depth in Surgical Vision Foundation Models: An Empirical Study of RGB-D Pre-training](https://mlsanity.com/paper/2601.18929/) (2026) - [GroundedSurg: A Multi-Procedure Benchmark for Language-Conditioned Surgical Tool Segmentation](https://mlsanity.com/paper/2603.01108/) (2026) - [RoboSurg-VQA: A Multimodal Benchmark for Surgical Segmentation-Aware Visual Question Answering](https://mlsanity.com/paper/2605.23068/) (2026) - [Stitch-Inferencer: Enhance Endoscopic Video Segmentation and Tracking via Panoramic Reconstruction](https://mlsanity.com/paper/2607.14968/) (2026) - [Topology-Aware Query Selection for Surgical Instrument Instance Segmentation](https://mlsanity.com/paper/2608.11607/) (2026) - [DART: Depth-as-Target Pretraining for Surgical Vision Foundation Models](https://mlsanity.com/paper/2609.04555/) (2026) - [Surgical Kinematics from Monocular Video with Learned Articulated Motion Constraints](https://mlsanity.com/paper/2609.27227/) (2026) ## Abstract (arXiv) In laparoscopic and robotic surgery, precise tool instance segmentation is an essential technology for advanced computer-assisted interventions. Although publicly available procedures of routine surgeries exist, they often lack comprehensive annotations for tool instance segmentation. Additionally, the majority of standard datasets for tool segmentation are derived from porcine(pig) surgeries. To address this gap, we introduce CholecInstanceSeg, the largest open-access tool instance segmentation dataset to date. Derived from the existing CholecT50 and Cholec80 datasets, CholecInstanceSeg provides novel annotations for laparoscopic cholecystectomy procedures in patients. Our dataset comprises 41.9k annotated frames extracted from 85 clinical procedures and 64.4k tool instances, each labelled with semantic masks and instance IDs. To ensure the reliability of our annotations, we perform extensive quality control, conduct label agreement statistics, and benchmark the segmentation results with various instance segmentation baselines. CholecInstanceSeg aims to advance the field by offering a comprehensive and high-quality open-access dataset for the development and evaluation of tool instance segmentation algorithms.