# L-MAE: Longitudinal masked auto-encoder with time and severity-aware encoding for diabetic retinopathy progression prediction

arXiv: https://arxiv.org/abs/2403.16272 · HTML: https://mlsanity.com/paper/2403.16272/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 55 references that are in the map.
- First posted on arXiv 24 Mar 2024.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [Sparse Sinkhorn Attention](https://mlsanity.com/paper/2002.11296/) (2020)
- [Transformers in Medical Imaging: A Survey](https://mlsanity.com/paper/2201.09873/) (2022)
- [A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trends](https://mlsanity.com/paper/2301.05712/) (2023)
- [CNN-RNN: A Unified Framework for Multi-label Image Classification](https://mlsanity.com/paper/1604.04573/) (2016)
- [Latent ODEs for Irregularly-Sampled Time Series](https://mlsanity.com/paper/1907.03907/) (2019)
- [On The Computational Complexity of Self-Attention](https://mlsanity.com/paper/2209.04881/) (2022)
- [Learning from Irregularly-Sampled Time Series: A Missing Data Perspective](https://mlsanity.com/paper/2008.07599/) (2020)
- [Lesion-based Contrastive Learning for Diabetic Retinopathy Grading from Fundus Images](https://mlsanity.com/paper/2107.08274/) (2021)
- [Improving Masked Autoencoders by Learning Where to Mask](https://mlsanity.com/paper/2303.06583/) (2023)
- [Transformer Acceleration with Dynamic Sparse Attention](https://mlsanity.com/paper/2110.11299/) (2021)
- [Metadata-enhanced contrastive learning from retinal optical coherence tomography images](https://mlsanity.com/paper/2208.02529/) (2022)
- [Time-distance vision transformers in lung cancer diagnosis from longitudinal computed tomography](https://mlsanity.com/paper/2209.01676/) (2022)
- [Predicting Risk of Developing Diabetic Retinopathy using Deep Learning](https://mlsanity.com/paper/2008.04370/) (2020)
- [LMT: Longitudinal Mixing Training, a Framework to Predict Disease Progression from a Single Image](https://mlsanity.com/paper/2310.10420/) (2023)
- [3DTINC: Time-Equivariant Non-Contrastive Learning for Predicting Disease Progression from Longitudinal OCTs](https://mlsanity.com/paper/2312.16980/) (2023)
- [DISCOVER: 2-D Multiview Summarization of Optical Coherence Tomography Angiography for Automatic Diabetic Retinopathy Diagnosis](https://mlsanity.com/paper/2401.05137/) (2024)
- [Development and Validation of a Novel Prognostic Model for Predicting AMD Progression Using Longitudinal Fundus Images](https://mlsanity.com/paper/2007.05120/) (2020)
- [Medical supervised masked autoencoders: Crafting a better masking strategy and efficient fine-tuning schedule for medical image classification](https://mlsanity.com/paper/2305.05871/) (2023)

## Built on by (most cited first, 2 of 2)

- [Stochastic Siamese MAE Pretraining for Longitudinal Medical Images](https://mlsanity.com/paper/2512.23441/) (2025)
- [Deep Learning for Longitudinal Medical Imaging: A Scoping Review](https://mlsanity.com/paper/2610.08838/) (2026)

## Abstract (arXiv)

Pre-training strategies based on self-supervised learning (SSL) have proven to be effective pretext tasks for many downstream tasks in computer vision. Due to the significant disparity between medical and natural images, the application of typical SSL is not straightforward in medical imaging. Additionally, those pretext tasks often lack context, which is critical for computer-aided clinical decision support. In this paper, we developed a longitudinal masked auto-encoder (MAE) based on the well-known Transformer-based MAE. In particular, we explored the importance of time-aware position embedding as well as disease progression-aware masking. Taking into account the time between examinations instead of just scheduling them offers the benefit of capturing temporal changes and trends. The masking strategy, for its part, evolves during follow-up to better capture pathological changes, ensuring a more accurate assessment of disease progression. Using OPHDIAT, a large follow-up screening dataset targeting diabetic retinopathy (DR), we evaluated the pre-trained weights on a longitudinal task, which is to predict the severity label of the next visit within 3 years based on the past time series examinations. Our results demonstrated the relevancy of both time-aware position embedding and masking strategies based on disease progression knowledge. Compared to popular baseline models and standard longitudinal Transformers, these simple yet effective extensions significantly enhance the predictive ability of deep classification models.