# Self-supervised dense representation learning for live-cell microscopy with time arrow prediction

arXiv: https://arxiv.org/abs/2305.05511 · HTML: https://mlsanity.com/paper/2305.05511/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 33 references that are in the map.
- First posted on arXiv 9 May 2023.

## Builds on (most cited first, 17 of 17)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [Context Encoders: Feature Learning by Inpainting](https://mlsanity.com/paper/1604.07379/) (2016)
- [Deep Sets](https://mlsanity.com/paper/1703.06114/) (2017)
- [Unsupervised Representation Learning by Sorting Sequences](https://mlsanity.com/paper/1708.01246/) (2017)
- [On Feature Decorrelation in Self-Supervised Learning](https://mlsanity.com/paper/2105.00470/) (2021)
- [Self-Supervised Representation Learning: Introduction, Advances and Challenges](https://mlsanity.com/paper/2110.09327/) (2021)
- [Self-Supervised Learning for Videos: A Survey](https://mlsanity.com/paper/2207.00419/) (2022)
- [Star-convex Polyhedra for 3D Object Detection and Segmentation in Microscopy](https://mlsanity.com/paper/1908.03636/) (2019)
- [Capturing implicit hierarchical structure in 3D biomedical images with self-supervised hyperbolic representations](https://mlsanity.com/paper/2012.01644/) (2020)
- [SCVRL: Shuffled Contrastive Video Representation Learning](https://mlsanity.com/paper/2205.11710/) (2022)

## Built on by (most cited first, 3 of 3)

- [DynaCLR: Contrastive Learning of Cellular Dynamics with Temporal Regularization](https://mlsanity.com/paper/2410.11281/) (2024)
- [Self-supervised Representation Learning for Cell Event Recognition through Time Arrow Prediction](https://mlsanity.com/paper/2411.03924/) (2024)
- [Unsupervised Learning of Cell Instances with Generative Routing Pyramids](https://mlsanity.com/paper/2608.16810/) (2026)

## Abstract (arXiv)

State-of-the-art object detection and segmentation methods for microscopy images rely on supervised machine learning, which requires laborious manual annotation of training data. Here we present a self-supervised method based on time arrow prediction pre-training that learns dense image representations from raw, unlabeled live-cell microscopy videos. Our method builds upon the task of predicting the correct order of time-flipped image regions via a single-image feature extractor followed by a time arrow prediction head that operates on the fused features. We show that the resulting dense representations capture inherently time-asymmetric biological processes such as cell divisions on a pixel-level. We furthermore demonstrate the utility of these representations on several live-cell microscopy datasets for detection and segmentation of dividing cells, as well as for cell state classification. Our method outperforms supervised methods, particularly when only limited ground truth annotations are available as is commonly the case in practice. We provide code at https://github.com/weigertlab/tarrow.