# Towards Real-Time Autonomous Navigation: Transformer-Based Catheter Tip Tracking in Fluoroscopy

arXiv: https://arxiv.org/abs/2605.14253 · HTML: https://mlsanity.com/paper/2605.14253/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 26 references that are in the map.
- First posted on arXiv 14 May 2026.

## Builds on (most cited first, 13 of 13)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation](https://mlsanity.com/paper/2105.05537/) (2021)
- [SegViT: Semantic Segmentation with Plain Vision Transformers](https://mlsanity.com/paper/2210.05844/) (2022)
- [End-to-End Real-time Catheter Segmentation with Optical Flow-Guided Warping during Endovascular Intervention](https://mlsanity.com/paper/2006.09117/) (2020)
- [Fully Automatic and Real-Time Catheter Segmentation in X-Ray Fluoroscopy](https://mlsanity.com/paper/1707.05137/) (2017)
- [CathSim: An Open-source Simulator for Endovascular Intervention](https://mlsanity.com/paper/2208.01455/) (2022)
- [Deep Segmentation and Registration in X-Ray Angiography Video](https://mlsanity.com/paper/1805.06406/) (2018)
- [Artificial Intelligence in the Autonomous Navigation of Endovascular Interventions: A Systematic Review](https://mlsanity.com/paper/2405.03305/) (2024)
- [Autonomous navigation of catheters and guidewires in mechanical thrombectomy using inverse reinforcement learning](https://mlsanity.com/paper/2406.12499/) (2024)
- [CathAction: A Benchmark for Endovascular Intervention Understanding](https://mlsanity.com/paper/2408.13126/) (2024)
- [Reinforcement Learning for Safe Autonomous Two Device Navigation of Cerebral Vessels in Mechanical Thrombectomy](https://mlsanity.com/paper/2503.24140/) (2025)
- [ConTrack: Contextual Transformer for Device Tracking in X-ray](https://mlsanity.com/paper/2307.07541/) (2023)

## Built on by (most cited first, 1 of 1)

- [Dual-Part Multi-Lateral Branched Network for Multi-Class Segmentation in Cardiovascular Catheterization Angiograms](https://mlsanity.com/paper/2609.04590/) (2026)

## Abstract (arXiv)

Purpose: Mechanical thrombectomy (MT) improves stroke outcomes, but is limited by a lack of local treatment access. Widespread distribution of reinforcement learning (RL)-based robotic systems can be used to alleviate this challenge through autonomous navigation, but current RL methods require live device tip coordinate tracking to function. This paper aims to develop and evaluate a real-time catheter tip tracking pipeline under fluoroscopy, addressing challenges such as low contrast, noise, and device occlusion. Methods: A multi-threaded pipeline was designed, incorporating frame reading, preprocessing, inference, and post-processing. Deep learning segmentation models, including U-Net, U-Net+Transformer, and SegFormer, were trained and benchmarked using two-class and three-class formulations. Post-processing involved two-step component filtering, one-pixel medial skeletonization, and greedy arc-length path following with contour fall-back. Results: On manually-labeled moderate complexity fluoroscopic video data, the two-class SegFormer achieved a mean absolute error of 4.44 mm, outperforming U-Net (4.60 mm), U-Net+Transformer (6.20 mm) and all three-class models (5.19-7.74 mm). On segmentation benchmarks, the system exceeded state-of-the-art CathAction results with improvements of up to +5% in Dice scores for three-segmentation. Conclusion: The results demonstrate that the proposed multi-threaded tracking framework maintains stable performance under challenging imaging conditions, outperforming prior benchmarks, while providing a reliable and efficient foundation for RL-based autonomous MT navigation.