# Trackastra: Transformer-based cell tracking for live-cell microscopy

arXiv: https://arxiv.org/abs/2405.15700 · HTML: https://mlsanity.com/paper/2405.15700/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 60 references that are in the map.
- First posted on arXiv 24 May 2024.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [LightGlue: Local Feature Matching at Light Speed](https://mlsanity.com/paper/2306.13643/) (2023)
- [TrackFormer: Multi-Object Tracking with Transformers](https://mlsanity.com/paper/2101.02702/) (2021)
- [Transformer Tracking](https://mlsanity.com/paper/2103.15436/) (2021)
- [MOTR: End-to-End Multiple-Object Tracking with Transformer](https://mlsanity.com/paper/2105.03247/) (2021)
- [TransTrack: Multiple Object Tracking with Transformer](https://mlsanity.com/paper/2012.15460/) (2020)
- [Cell Detection with Star-convex Polygons](https://mlsanity.com/paper/1806.03535/) (2018)
- [Multiple Object Tracking: A Literature Review](https://mlsanity.com/paper/1409.7618/) (2014)
- [MOTChallenge 2015: Towards a Benchmark for Multi-Target Tracking](https://mlsanity.com/paper/1504.01942/) (2015)
- [Global Tracking Transformers](https://mlsanity.com/paper/2203.13250/) (2022)
- [TransMOT: Spatial-Temporal Graph Transformer for Multiple Object Tracking](https://mlsanity.com/paper/2104.00194/) (2021)
- [Deep Network Flow for Multi-Object Tracking](https://mlsanity.com/paper/1706.08482/) (2017)
- [Transformers in 3D Point Clouds: A Survey](https://mlsanity.com/paper/2205.07417/) (2022)
- [Graph Neural Network for Cell Tracking in Microscopy Videos](https://mlsanity.com/paper/2202.04731/) (2022)
- [EmbedTrack -- Simultaneous Cell Segmentation and Tracking Through Learning Offsets and Clustering Bandwidths](https://mlsanity.com/paper/2204.10713/) (2022)
- [MPM: Joint Representation of Motion and Position Map for Cell Tracking](https://mlsanity.com/paper/2002.10749/) (2020)
- [NOVIS: A Case for End-to-End Near-Online Video Instance Segmentation](https://mlsanity.com/paper/2308.15266/) (2023)
- [Large-Scale Multi-Hypotheses Cell Tracking Using Ultrametric Contours Maps](https://mlsanity.com/paper/2308.04526/) (2023)
- [Moral Lineage Tracing](https://mlsanity.com/paper/1511.05512/) (2015)
- [Tracking by weakly-supervised learning and graph optimization for whole-embryo C. elegans lineages](https://mlsanity.com/paper/2208.11467/) (2022)

## Built on by (most cited first, 14 of 14)

- [Tracking one-in-a-million: Large-scale benchmark for microbial single-cell tracking with experiment-aware robustness metrics](https://mlsanity.com/paper/2411.00552/) (2024)
- [How To Make Your Cell Tracker Say "I dunno!"](https://mlsanity.com/paper/2503.09244/) (2025)
- [PyUAT: Open-source Python framework for efficient and scalable cell tracking](https://mlsanity.com/paper/2503.21914/) (2025)
- [Segment Anything for Cell Tracking](https://mlsanity.com/paper/2509.09943/) (2025)
- [TransientTrack: Advanced Multi-Object Tracking and Classification of Cancer Cells with Transient Fluorescent Signals](https://mlsanity.com/paper/2512.01885/) (2025)
- [Cell as Point: One-Stage Framework for Efficient Cell Tracking](https://mlsanity.com/paper/2411.14833/) (2024)
- [WaveOrder: A differentiable wave-optical framework for scalable biological microscopy with diverse modalities](https://mlsanity.com/paper/2412.09775/) (2024)
- [Attention-Bayesian Hybrid Approach to Modular Multiple Particle Tracking](https://mlsanity.com/paper/2506.09441/) (2025)
- [acia-workflows: Automated Single-cell Imaging Analysis for Scalable and Deep Learning-based Live-cell Imaging Analysis Workflows](https://mlsanity.com/paper/2510.05886/) (2025)
- [MicroscopyMatching: Towards a Ready-to-use Framework for Microscopy Image Analysis in Diverse Conditions](https://mlsanity.com/paper/2605.14980/) (2026)
- [Automated Erythrocyte Detection and Tracking for Retinal Blood Flow Quantification in Erythrocyte-Mediated Angiography](https://mlsanity.com/paper/2606.01006/) (2026)
- [ARGUS: Accelerated, Robust, General, and Unsupervised Cell Tracking Solutions](https://mlsanity.com/paper/2607.08297/) (2026)
- [Higher-Order Cell Tracking Transformer](https://mlsanity.com/paper/2607.11754/) (2026)
- [SAM+D: Parameter-Efficient Dimensional Lifting of SAM-Family Models via Depth-Routed LoRA and Depth Shifting](https://mlsanity.com/paper/2607.29033/) (2026)

## Abstract (arXiv)

Cell tracking is a ubiquitous image analysis task in live-cell microscopy. Unlike multiple object tracking (MOT) for natural images, cell tracking typically involves hundreds of similar-looking objects that can divide in each frame, making it a particularly challenging problem. Current state-of-the-art approaches follow the tracking-by-detection paradigm, i.e. first all cells are detected per frame and successively linked in a second step to form biologically consistent cell tracks. Linking is commonly solved via discrete optimization methods, which require manual tuning of hyperparameters for each dataset and are therefore cumbersome to use in practice. Here we propose Trackastra, a general purpose cell tracking approach that uses a simple transformer architecture to directly learn pairwise associations of cells within a temporal window from annotated data. Importantly, unlike existing transformer-based MOT pipelines, our learning architecture also accounts for dividing objects such as cells and allows for accurate tracking even with simple greedy linking, thus making strides towards removing the requirement for a complex linking step. The proposed architecture operates on the full spatio-temporal context of detections within a time window by avoiding the computational burden of processing dense images. We show that our tracking approach performs on par with or better than highly tuned state-of-the-art cell tracking algorithms for various biological datasets, such as bacteria, cell cultures and fluorescent particles. We provide code at https://github.com/weigertlab/trackastra.