# Surgical Tattoos in Infrared: A Dataset for Quantifying Tissue Tracking and Mapping

arXiv: https://arxiv.org/abs/2309.16782 · HTML: https://mlsanity.com/paper/2309.16782/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 44 references that are in the map.
- First posted on arXiv 28 Sep 2023.

## Builds on (most cited first, 21 of 21)

- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [PointOdyssey: A Large-Scale Synthetic Dataset for Long-Term Point Tracking](https://mlsanity.com/paper/2307.15055/) (2023)
- [TAP-Vid: A Benchmark for Tracking Any Point in a Video](https://mlsanity.com/paper/2211.03726/) (2022)
- [Particle Video Revisited: Tracking Through Occlusions Using Point Trajectories](https://mlsanity.com/paper/2204.04153/) (2022)
- [Metrics reloaded: Recommendations for image analysis validation](https://mlsanity.com/paper/2206.01653/) (2022)
- [Tracking Everything Everywhere All at Once](https://mlsanity.com/paper/2306.05422/) (2023)
- [The VIA Annotation Software for Images, Audio and Video](https://mlsanity.com/paper/1904.10699/) (2019)
- [Neural Rendering for Stereo 3D Reconstruction of Deformable Tissues in Robotic Surgery](https://mlsanity.com/paper/2206.15255/) (2022)
- [Stereo Correspondence and Reconstruction of Endoscopic Data Challenge](https://mlsanity.com/paper/2101.01133/) (2021)
- [Endo-Depth-and-Motion: Reconstruction and Tracking in Endoscopic Videos using Depth Networks and Photometric Constraints](https://mlsanity.com/paper/2103.16525/) (2021)
- [Colonoscopy 3D Video Dataset with Paired Depth from 2D-3D Registration](https://mlsanity.com/paper/2206.08903/) (2022)
- [MFT: Long-Term Tracking of Every Pixel](https://mlsanity.com/paper/2305.12998/) (2023)
- [SuPer: A Surgical Perception Framework for Endoscopic Tissue Manipulation with Surgical Robotics](https://mlsanity.com/paper/1909.05405/) (2019)
- [CholecTriplet2021: A benchmark challenge for surgical action triplet recognition](https://mlsanity.com/paper/2204.04746/) (2022)
- [SERV-CT: A disparity dataset from CT for validation of endoscopic 3D reconstruction](https://mlsanity.com/paper/2012.11779/) (2020)
- [Semantic-SuPer: A Semantic-aware Surgical Perception Framework for Endoscopic Tissue Identification, Reconstruction, and Tracking](https://mlsanity.com/paper/2210.16674/) (2022)
- [SurgT challenge: Benchmark of Soft-Tissue Trackers for Robotic Surgery](https://mlsanity.com/paper/2302.03022/) (2023)
- [SENDD: Sparse Efficient Neural Depth and Deformation for Tissue Tracking](https://mlsanity.com/paper/2305.06477/) (2023)
- [Lighting Enhancement Aids Reconstruction of Colonoscopic Surfaces](https://mlsanity.com/paper/2103.10310/) (2021)
- [Direct and Sparse Deformable Tracking](https://mlsanity.com/paper/2109.07370/) (2021)
- [Autonomous Scanning for Endomicroscopic Mosaicing and 3D Fusion](https://mlsanity.com/paper/1604.04137/) (2016)

## Built on by (most cited first, 15 of 15)

- [BootsTAP: Bootstrapped Training for Tracking-Any-Point](https://mlsanity.com/paper/2402.00847/) (2024)
- [TAPNext: Tracking Any Point (TAP) as Next Token Prediction](https://mlsanity.com/paper/2504.05579/) (2025)
- [SurgPose: a Dataset for Articulated Robotic Surgical Tool Pose Estimation and Tracking](https://mlsanity.com/paper/2502.11534/) (2025)
- [Point Tracking in Surgery--The 2024 Surgical Tattoos in Infrared (STIR) Challenge](https://mlsanity.com/paper/2503.24306/) (2025)
- [SciVid: Cross-Domain Evaluation of Video Models in Scientific Applications](https://mlsanity.com/paper/2507.03578/) (2025)
- [Tracking Everything in Robotic-Assisted Surgery](https://mlsanity.com/paper/2409.19821/) (2024)
- [LiteTracker: Leveraging Temporal Causality for Accurate Low-latency Tissue Tracking](https://mlsanity.com/paper/2504.09904/) (2025)
- [BleedOrigin: Dynamic Bleeding Source Localization in Endoscopic Submucosal Dissection via Dual-Stage Detection and Tracking](https://mlsanity.com/paper/2507.15094/) (2025)
- [Endo-TTAP: Robust Endoscopic Tissue Tracking via Multi-Facet Guided Attention and Hybrid Flow-point Supervision](https://mlsanity.com/paper/2503.22394/) (2025)
- [AutoCam: Hierarchical Path Planning for an Autonomous Auxiliary Camera in Surgical Robotics](https://mlsanity.com/paper/2505.10398/) (2025)
- [Surg-InvNeRF: Invertible NeRF for 3D tracking and reconstruction in surgical vision](https://mlsanity.com/paper/2508.09681/) (2025)
- [Bridging Vision and Language for Robust Context-Aware Surgical Point Tracking: The VL-SurgPT Dataset and Benchmark](https://mlsanity.com/paper/2511.12026/) (2025)
- [Point Tracking in Surgery--The 2025 Surgical Tattoos in Infrared Challenge (STIRC2025)](https://mlsanity.com/paper/2607.12939/) (2026)
- [Stitch-Inferencer: Enhance Endoscopic Video Segmentation and Tracking via Panoramic Reconstruction](https://mlsanity.com/paper/2607.14968/) (2026)
- [S3-Tracker: Self-Supervised Surgical Tissue Tracking With Contrastive Random Walks](https://mlsanity.com/paper/2609.14313/) (2026)

## Abstract (arXiv)

Quantifying performance of methods for tracking and mapping tissue in endoscopic environments is essential for enabling image guidance and automation of medical interventions and surgery. Datasets developed so far either use rigid environments, visible markers, or require annotators to label salient points in videos after collection. These are respectively: not general, visible to algorithms, or costly and error-prone. We introduce a novel labeling methodology along with a dataset that uses said methodology, Surgical Tattoos in Infrared (STIR). STIR has labels that are persistent but invisible to visible spectrum algorithms. This is done by labelling tissue points with IR-fluorescent dye, indocyanine green (ICG), and then collecting visible light video clips. STIR comprises hundreds of stereo video clips in both in-vivo and ex-vivo scenes with start and end points labelled in the IR spectrum. With over 3,000 labelled points, STIR will help to quantify and enable better analysis of tracking and mapping methods. After introducing STIR, we analyze multiple different frame-based tracking methods on STIR using both 3D and 2D endpoint error and accuracy metrics. STIR is available at https://dx.doi.org/10.21227/w8g4-g548