# SurgPose: a Dataset for Articulated Robotic Surgical Tool Pose Estimation and Tracking

arXiv: https://arxiv.org/abs/2502.11534 · HTML: https://mlsanity.com/paper/2502.11534/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 47 references that are in the map.
- First posted on arXiv 17 Feb 2025.

## Builds on (most cited first, 23 of 23)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [Planning-oriented Autonomous Driving](https://mlsanity.com/paper/2212.10156/) (2022)
- [OpenScene: 3D Scene Understanding with Open Vocabularies](https://mlsanity.com/paper/2211.15654/) (2022)
- [ViTPose: Simple Vision Transformer Baselines for Human Pose Estimation](https://mlsanity.com/paper/2204.12484/) (2022)
- [Markerless tracking of user-defined features with deep learning](https://mlsanity.com/paper/1804.03142/) (2018)
- [PoseTrack: A Benchmark for Human Pose Estimation and Tracking](https://mlsanity.com/paper/1710.10000/) (2017)
- [APT-36K: A Large-scale Benchmark for Animal Pose Estimation and Tracking](https://mlsanity.com/paper/2206.05683/) (2022)
- [Surgical Robot Transformer (SRT): Imitation Learning for Surgical Tasks](https://mlsanity.com/paper/2407.12998/) (2024)
- [Comparative evaluation of instrument segmentation and tracking methods in minimally invasive surgery](https://mlsanity.com/paper/1805.02475/) (2018)
- [Pose Estimation for Robot Manipulators via Keypoint Optimization and Sim-to-Real Transfer](https://mlsanity.com/paper/2010.08054/) (2020)
- [Surgical Tattoos in Infrared: A Dataset for Quantifying Tissue Tracking and Mapping](https://mlsanity.com/paper/2309.16782/) (2023)
- [Simultaneous Recognition and Pose Estimation of Instruments in Minimally Invasive Surgery](https://mlsanity.com/paper/1710.06668/) (2017)
- [SENDD: Sparse Efficient Neural Depth and Deformation for Tissue Tracking](https://mlsanity.com/paper/2305.06477/) (2023)
- [Real-time 3D Tracking of Articulated Tools for Robotic Surgery](https://mlsanity.com/paper/1605.03483/) (2016)
- [Tracking Everything in Robotic-Assisted Surgery](https://mlsanity.com/paper/2409.19821/) (2024)
- [Deep Attention Based Semi-Supervised 2D-Pose Estimation for Surgical Instruments](https://mlsanity.com/paper/1912.04618/) (2019)
- [All You Need is LUV: Unsupervised Collection of Labeled Images using Invisible UV Fluorescent Indicators](https://mlsanity.com/paper/2203.04566/) (2022)
- [Arc-to-line frame registration method for ultrasound and photoacoustic image-guided intraoperative robot-assisted laparoscopic prostatectomy](https://mlsanity.com/paper/2306.12590/) (2023)
- [Automatic Search for Photoacoustic Marker Using Automated Transrectal Ultrasound](https://mlsanity.com/paper/2307.10546/) (2023)

## Built on by (most cited first, 16 of 16)

- [Instrument-Splatting: Controllable Photorealistic Reconstruction of Surgical Instruments Using Gaussian Splatting](https://mlsanity.com/paper/2503.04082/) (2025)
- [SutureBot: A Precision Framework & Benchmark For Autonomous End-to-End Suturing](https://mlsanity.com/paper/2510.20965/) (2025)
- [ROBUST-MIPS: A Combined Skeletal Pose and Instance Segmentation Dataset for Laparoscopic Surgical Instruments](https://mlsanity.com/paper/2508.21096/) (2025)
- [ToolTipNet: A Segmentation-Driven Deep Learning Baseline for Surgical Instrument Tip Detection](https://mlsanity.com/paper/2504.09700/) (2025)
- [AutoCam: Hierarchical Path Planning for an Autonomous Auxiliary Camera in Surgical Robotics](https://mlsanity.com/paper/2505.10398/) (2025)
- [Estimating 2D Keypoints of Surgical Tools Using Vision-Language Models with Low-Rank Adaptation](https://mlsanity.com/paper/2508.20830/) (2025)
- [SurgCalib: Gaussian Splatting-Based Hand-Eye Calibration for Robot-Assisted Minimally Invasive Surgery](https://mlsanity.com/paper/2603.08983/) (2026)
- [Real-time Rendering-based Surgical Instrument Tracking via Evolutionary Optimization](https://mlsanity.com/paper/2603.11404/) (2026)
- [The Blind Spot in 2D Infants' Pose Estimation:Robust Learning from Noisy Annotations](https://mlsanity.com/paper/2609.04009/) (2026)
- [On the Role of Depth in Surgical Vision Foundation Models: An Empirical Study of RGB-D Pre-training](https://mlsanity.com/paper/2601.18929/) (2026)
- [Training-free Detection and 6D Pose Estimation of Unseen Surgical Instruments](https://mlsanity.com/paper/2603.25228/) (2026)
- [Exploring 6D Object Pose Estimation with Deformation](https://mlsanity.com/paper/2604.06720/) (2026)
- [SurfSurg6D: Geometry Consistent Dense Correspondence for Textureless Surgical Instrument Pose Estimation](https://mlsanity.com/paper/2605.25598/) (2026)
- [Dense Structural Priors for Sparse Functional Landmark Localization in Surgical Videos](https://mlsanity.com/paper/2606.31007/) (2026)
- [Robust Surgical Robotic Instrument Tracking via Sequential Multi-Cue Fusion and Sim-to-Real Self-Training](https://mlsanity.com/paper/2610.05491/) (2026)
- [SurgCast: Action-Conditioned Future Skeletons for Controllable Surgical World Models](https://mlsanity.com/paper/2610.15645/) (2026)

## Abstract (arXiv)

Accurate and efficient surgical robotic tool pose estimation is of fundamental significance to downstream applications such as augmented reality (AR) in surgical training and learning-based autonomous manipulation. While significant advancements have been made in pose estimation for humans and animals, it is still a challenge in surgical robotics due to the scarcity of published data. The relatively large absolute error of the da Vinci end effector kinematics and arduous calibration procedure make calibrated kinematics data collection expensive. Driven by this limitation, we collected a dataset, dubbed SurgPose, providing instance-aware semantic keypoints and skeletons for visual surgical tool pose estimation and tracking. By marking keypoints using ultraviolet (UV) reactive paint, which is invisible under white light and fluorescent under UV light, we execute the same trajectory under different lighting conditions to collect raw videos and keypoint annotations, respectively. The SurgPose dataset consists of approximately 120k surgical instrument instances (80k for training and 40k for validation) of 6 categories. Each instrument instance is labeled with 7 semantic keypoints. Since the videos are collected in stereo pairs, the 2D pose can be lifted to 3D based on stereo-matching depth. In addition to releasing the dataset, we test a few baseline approaches to surgical instrument tracking to demonstrate the utility of SurgPose. More details can be found at surgpose.github.io.