# Self-Supervised Learning for Interventional Image Analytics: Towards Robust Device Trackers

arXiv: https://arxiv.org/abs/2405.01156 · HTML: https://mlsanity.com/paper/2405.01156/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 74 references that are in the map.
- First posted on arXiv 2 May 2024.

## Builds on (most cited first, 30 of 41)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training](https://mlsanity.com/paper/2203.12602/) (2022)
- [SimMIM: A Simple Framework for Masked Image Modeling](https://mlsanity.com/paper/2111.09886/) (2021)
- [Unsupervised Visual Representation Learning by Context Prediction](https://mlsanity.com/paper/1505.05192/) (2015)
- [Time-Contrastive Networks: Self-Supervised Learning from Video](https://mlsanity.com/paper/1704.06888/) (2017)
- [SiamRPN++: Evolution of Siamese Visual Tracking with Very Deep Networks](https://mlsanity.com/paper/1812.11703/) (2018)
- [LaSOT: A High-quality Benchmark for Large-scale Single Object Tracking](https://mlsanity.com/paper/1809.07845/) (2018)
- [Masked Autoencoders As Spatiotemporal Learners](https://mlsanity.com/paper/2205.09113/) (2022)
- [Transformer Tracking](https://mlsanity.com/paper/2103.15436/) (2021)
- [Spatiotemporal Contrastive Video Representation Learning](https://mlsanity.com/paper/2008.03800/) (2020)
- [Learning Spatio-Temporal Transformer for Visual Tracking](https://mlsanity.com/paper/2103.17154/) (2021)
- [Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolation](https://mlsanity.com/paper/1712.00080/) (2017)
- [MixFormer: End-to-End Tracking with Iterative Mixed Attention](https://mlsanity.com/paper/2203.11082/) (2022)
- [Transformer Meets Tracker: Exploiting Temporal Context for Robust Visual Tracking](https://mlsanity.com/paper/2103.11681/) (2021)
- [A Large-Scale Study on Unsupervised Spatiotemporal Representation Learning](https://mlsanity.com/paper/2104.14558/) (2021)
- [Video Frame Interpolation via Adaptive Separable Convolution](https://mlsanity.com/paper/1708.01692/) (2017)
- [Distractor-aware Siamese Networks for Visual Object Tracking](https://mlsanity.com/paper/1808.06048/) (2018)
- [Video Representation Learning by Dense Predictive Coding](https://mlsanity.com/paper/1909.04656/) (2019)

## Built on by (most cited first, 1 of 1)

- [A Novel Tracking Framework for Devices in X-ray Leveraging Supplementary Cue-Driven Self-Supervised Features](https://mlsanity.com/paper/2501.12958/) (2025)

## Abstract (arXiv)

An accurate detection and tracking of devices such as guiding catheters in live X-ray image acquisitions is an essential prerequisite for endovascular cardiac interventions. This information is leveraged for procedural guidance, e.g., directing stent placements. To ensure procedural safety and efficacy, there is a need for high robustness no failures during tracking. To achieve that, one needs to efficiently tackle challenges, such as: device obscuration by contrast agent or other external devices or wires, changes in field-of-view or acquisition angle, as well as the continuous movement due to cardiac and respiratory motion. To overcome the aforementioned challenges, we propose a novel approach to learn spatio-temporal features from a very large data cohort of over 16 million interventional X-ray frames using self-supervision for image sequence data. Our approach is based on a masked image modeling technique that leverages frame interpolation based reconstruction to learn fine inter-frame temporal correspondences. The features encoded in the resulting model are fine-tuned downstream. Our approach achieves state-of-the-art performance and in particular robustness compared to ultra optimized reference solutions (that use multi-stage feature fusion, multi-task and flow regularization). The experiments show that our method achieves 66.31% reduction in maximum tracking error against reference solutions (23.20% when flow regularization is used); achieving a success score of 97.95% at a 3x faster inference speed of 42 frames-per-second (on GPU). The results encourage the use of our approach in various other tasks within interventional image analytics that require effective understanding of spatio-temporal semantics.