# Local Spatiotemporal Representation Learning for Longitudinally-consistent Neuroimage Analysis

arXiv: https://arxiv.org/abs/2206.04281 · HTML: https://mlsanity.com/paper/2206.04281/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 90 references that are in the map.
- First posted on arXiv 9 Jun 2022.

## Builds on (most cited first, 30 of 43)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [Context Encoders: Feature Learning by Inpainting](https://mlsanity.com/paper/1604.07379/) (2016)
- [Big Self-Supervised Models are Strong Semi-Supervised Learners](https://mlsanity.com/paper/2006.10029/) (2020)
- [Contrastive Multiview Coding](https://mlsanity.com/paper/1906.05849/) (2019)
- [Learning deep representations by mutual information estimation and maximization](https://mlsanity.com/paper/1808.06670/) (2018)
- [VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning](https://mlsanity.com/paper/2105.04906/) (2021)
- [Data-Efficient Image Recognition with Contrastive Predictive Coding](https://mlsanity.com/paper/1905.09272/) (2019)
- [Contrastive Learning for Unpaired Image-to-Image Translation](https://mlsanity.com/paper/2007.15651/) (2020)
- [Understanding Dimensional Collapse in Contrastive Self-supervised Learning](https://mlsanity.com/paper/2110.09348/) (2021)
- [3D MRI brain tumor segmentation using autoencoder regularization](https://mlsanity.com/paper/1810.11654/) (2018)
- [Contrastive learning of global and local features for medical image segmentation with limited annotations](https://mlsanity.com/paper/2006.10511/) (2020)
- [Exploring Cross-Image Pixel Contrast for Semantic Segmentation](https://mlsanity.com/paper/2101.11939/) (2021)
- [Split-Brain Autoencoders: Unsupervised Learning by Cross-Channel Prediction](https://mlsanity.com/paper/1611.09842/) (2016)
- [TorchIO: A Python library for efficient loading, preprocessing, augmentation and patch-based sampling of medical images in deep learning](https://mlsanity.com/paper/2003.04696/) (2020)
- [Self-supervised Video Representation Learning by Pace Prediction](https://mlsanity.com/paper/2008.05861/) (2020)
- [Semi-Supervised Semantic Segmentation with Pixel-Level Contrastive Learning from a Class-wise Memory Bank](https://mlsanity.com/paper/2104.13415/) (2021)
- [Boosting Contrastive Self-Supervised Learning with False Negative Cancellation](https://mlsanity.com/paper/2011.11765/) (2020)
- [Pixel Contrastive-Consistent Semi-Supervised Semantic Segmentation](https://mlsanity.com/paper/2108.09025/) (2021)
- [Contrastive Learning for Label-Efficient Semantic Segmentation](https://mlsanity.com/paper/2012.06985/) (2020)
- [Bootstrapping Semantic Segmentation with Regional Contrast](https://mlsanity.com/paper/2104.04465/) (2021)
- [Connect, Not Collapse: Explaining Contrastive Learning for Unsupervised Domain Adaptation](https://mlsanity.com/paper/2204.00570/) (2022)

## Built on by (most cited first, 19 of 19)

- [Learning General-Purpose Biomedical Volume Representations using Randomized Synthesis](https://mlsanity.com/paper/2411.02372/) (2024)
- [Sequential Multi-Dimensional Self-Supervised Learning for Clinical Time Series](https://mlsanity.com/paper/2307.10923/) (2023)
- [Learning to Compare Longitudinal Images](https://mlsanity.com/paper/2304.02531/) (2023)
- [ContraReg: Contrastive Learning of Multi-modality Unsupervised Deformable Image Registration](https://mlsanity.com/paper/2206.13434/) (2022)
- [Learning Patient-Specific Disease Dynamics with Latent Flow Matching for Longitudinal Imaging Generation](https://mlsanity.com/paper/2512.09185/) (2025)
- [Keypoint-Augmented Self-Supervised Learning for Medical Image Segmentation with Limited Annotation](https://mlsanity.com/paper/2310.01680/) (2023)
- [Self-Supervised Alignment Learning for Medical Image Segmentation](https://mlsanity.com/paper/2406.15699/) (2024)
- [LaTiM: Longitudinal representation learning in continuous-time models to predict disease progression](https://mlsanity.com/paper/2404.07091/) (2024)
- [Longitudinal Self-supervised Learning Using Neural Ordinary Differential Equation](https://mlsanity.com/paper/2310.10431/) (2023)
- [TimeFlow: Temporal Conditioning for Longitudinal Brain MRI Registration and Aging Analysis](https://mlsanity.com/paper/2501.08667/) (2025)
- [Dual Meta-Learning with Longitudinally Generalized Regularization for One-Shot Brain Tissue Segmentation Across the Human Lifespan](https://mlsanity.com/paper/2308.06774/) (2023)
- [Spatiotemporal Representation Learning for Short and Long Medical Image Time Series](https://mlsanity.com/paper/2403.07513/) (2024)
- [Regional Deep Atrophy: a Self-Supervised Learning Method to Automatically Identify Regions Associated With Alzheimer's Disease Progression From Longitudinal MRI](https://mlsanity.com/paper/2304.04673/) (2023)
- [Consistency Regularization Improves Placenta Segmentation in Fetal EPI MRI Time Series](https://mlsanity.com/paper/2310.03870/) (2023)
- [Identity Preserving Latent Diffusion for Brain Aging Modeling](https://mlsanity.com/paper/2503.09634/) (2025)
- [Multimodal Disease Progression Modeling via Spatiotemporal Disentanglement and Multiscale Alignment](https://mlsanity.com/paper/2510.11112/) (2025)
- [Stochastic Siamese MAE Pretraining for Longitudinal Medical Images](https://mlsanity.com/paper/2512.23441/) (2025)
- [V2C-Long: Longitudinal Cortex Reconstruction with Spatiotemporal Correspondence](https://mlsanity.com/paper/2402.17438/) (2024)
- [Anatomy-DT: A Cross-Diffusion Digital Twin for Anatomical Evolution](https://mlsanity.com/paper/2509.25280/) (2025)

## Abstract (arXiv)

Recent self-supervised advances in medical computer vision exploit global and local anatomical self-similarity for pretraining prior to downstream tasks such as segmentation. However, current methods assume i.i.d. image acquisition, which is invalid in clinical study designs where follow-up longitudinal scans track subject-specific temporal changes. Further, existing self-supervised methods for medically-relevant image-to-image architectures exploit only spatial or temporal self-similarity and only do so via a loss applied at a single image-scale, with naive multi-scale spatiotemporal extensions collapsing to degenerate solutions. To these ends, this paper makes two contributions: (1) It presents a local and multi-scale spatiotemporal representation learning method for image-to-image architectures trained on longitudinal images. It exploits the spatiotemporal self-similarity of learned multi-scale intra-subject features for pretraining and develops several feature-wise regularizations that avoid collapsed identity representations; (2) During finetuning, it proposes a surprisingly simple self-supervised segmentation consistency regularization to exploit intra-subject correlation. Benchmarked in the one-shot segmentation setting, the proposed framework outperforms both well-tuned randomly-initialized baselines and current self-supervised techniques designed for both i.i.d. and longitudinal datasets. These improvements are demonstrated across both longitudinal neurodegenerative adult MRI and developing infant brain MRI and yield both higher performance and longitudinal consistency.