# Cell as Point: One-Stage Framework for Efficient Cell Tracking

arXiv: https://arxiv.org/abs/2411.14833 · HTML: https://mlsanity.com/paper/2411.14833/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 38 of its 93 references that are in the map.
- First posted on arXiv 22 Nov 2024.

## Builds on (most cited first, 30 of 38)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [FlowNet: Learning Optical Flow with Convolutional Networks](https://mlsanity.com/paper/1504.06852/) (2015)
- [PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume](https://mlsanity.com/paper/1709.02371/) (2017)
- [FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks](https://mlsanity.com/paper/1612.01925/) (2016)
- [CoTracker: It is Better to Track Together](https://mlsanity.com/paper/2307.07635/) (2023)
- [CRIS: CLIP-Driven Referring Image Segmentation](https://mlsanity.com/paper/2111.15174/) (2021)
- [PointOdyssey: A Large-Scale Synthetic Dataset for Long-Term Point Tracking](https://mlsanity.com/paper/2307.15055/) (2023)
- [TAP-Vid: A Benchmark for Tracking Any Point in a Video](https://mlsanity.com/paper/2211.03726/) (2022)
- [Particle Video Revisited: Tracking Through Occlusions Using Point Trajectories](https://mlsanity.com/paper/2204.04153/) (2022)
- [Tracking Everything Everywhere All at Once](https://mlsanity.com/paper/2306.05422/) (2023)
- [CoTracker3: Simpler and Better Point Tracking by Pseudo-Labelling Real Videos](https://mlsanity.com/paper/2410.11831/) (2024)
- [Towards An End-to-End Framework for Flow-Guided Video Inpainting](https://mlsanity.com/paper/2204.02663/) (2022)
- [FitVid: Overfitting in Pixel-Level Video Prediction](https://mlsanity.com/paper/2106.13195/) (2021)
- [Spatial-Temporal Relation Networks for Multi-Object Tracking](https://mlsanity.com/paper/1904.11489/) (2019)
- [Accurate Optical Flow via Direct Cost Volume Processing](https://mlsanity.com/paper/1704.07325/) (2017)
- [Unsupervised 3D End-to-End Medical Image Registration with Volume Tweening Network](https://mlsanity.com/paper/1902.05020/) (2019)
- [Star-convex Polyhedra for 3D Object Detection and Segmentation in Microscopy](https://mlsanity.com/paper/1908.03636/) (2019)
- [Multi-Class Cell Detection Using Spatial Context Representation](https://mlsanity.com/paper/2110.04886/) (2021)
- [Segment Any Cell: A SAM-based Auto-prompting Fine-tuning Framework for Nuclei Segmentation](https://mlsanity.com/paper/2401.13220/) (2024)
- [Graph Neural Network for Cell Tracking in Microscopy Videos](https://mlsanity.com/paper/2202.04731/) (2022)
- [EmbedTrack -- Simultaneous Cell Segmentation and Tracking Through Learning Offsets and Clustering Bandwidths](https://mlsanity.com/paper/2204.10713/) (2022)
- [Overfitting the Data: Compact Neural Video Delivery via Content-aware Feature Modulation](https://mlsanity.com/paper/2108.08202/) (2021)
- [Instance Segmentation and Tracking with Cosine Embeddings and Recurrent Hourglass Networks](https://mlsanity.com/paper/1806.02070/) (2018)
- [Trackastra: Transformer-based cell tracking for live-cell microscopy](https://mlsanity.com/paper/2405.15700/) (2024)
- [MPM: Joint Representation of Motion and Position Map for Cell Tracking](https://mlsanity.com/paper/2002.10749/) (2020)
- [CellTrack R-CNN: A Novel End-To-End Deep Neural Network for Cell Segmentation and Tracking in Microscopy Images](https://mlsanity.com/paper/2102.10377/) (2021)
- [OmniFlow: Human Omnidirectional Optical Flow](https://mlsanity.com/paper/2104.07960/) (2021)
- [Weakly-Supervised Cell Tracking via Backward-and-Forward Propagation](https://mlsanity.com/paper/2007.15258/) (2020)
- [J Regularization Improves Imbalanced Multiclass Segmentation](https://mlsanity.com/paper/1910.09783/) (2019)

## Abstract (arXiv)

Conventional multi-stage cell tracking approaches rely heavily on detection or segmentation in each frame as a prerequisite, requiring substantial resources for high-quality segmentation masks and increasing the overall prediction time. To address these limitations, we propose CAP, a novel end-to-end one-stage framework that reimagines cell tracking by treating Cell as Point. Unlike traditional methods, CAP eliminates the need for explicit detection or segmentation, instead jointly tracking cells for sequences in one stage by leveraging the inherent correlations among their trajectories. This simplification reduces both labeling requirements and pipeline complexity. However, directly processing the entire sequence in one stage poses challenges related to data imbalance in capturing cell division events and long sequence inference. To solve these challenges, CAP introduces two key innovations: (1) adaptive event-guided (AEG) sampling, which prioritizes cell division events to mitigate the occurrence imbalance of cell events, and (2) the rolling-as-window (RAW) inference strategy, which ensures continuous and stable tracking of newly emerging cells over extended sequences. By removing the dependency on segmentation-based preprocessing while addressing the challenges of imbalanced occurrence of cell events and long-sequence tracking, CAP demonstrates promising cell tracking performance and is 8 to 32 times more efficient than existing methods. The code and model checkpoints are available at https://github.com/YXSong000/CAP.