# ImageFlowNet: Forecasting Multiscale Image-Level Trajectories of Disease Progression with Irregularly-Sampled Longitudinal Medical Images

arXiv: https://arxiv.org/abs/2406.14794 · HTML: https://mlsanity.com/paper/2406.14794/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 53 of its 121 references that are in the map.
- First posted on arXiv 20 Jun 2024.

## Builds on (most cited first, 30 of 53)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Imagen Video: High Definition Video Generation with Diffusion Models](https://mlsanity.com/paper/2210.02303/) (2022)
- [Image Super-Resolution Using Deep Convolutional Networks](https://mlsanity.com/paper/1501.00092/) (2014)
- [A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges](https://mlsanity.com/paper/2011.06225/) (2020)
- [Albumentations: fast and flexible image augmentations](https://mlsanity.com/paper/1809.06839/) (2018)
- [Lumiere: A Space-Time Diffusion Model for Video Generation](https://mlsanity.com/paper/2401.12945/) (2024)
- [BBDM: Image-to-image Translation with Brownian Bridge Diffusion Models](https://mlsanity.com/paper/2205.07680/) (2022)
- [I2SB: Image-to-Image Schr"odinger Bridge](https://mlsanity.com/paper/2302.05872/) (2023)
- [A Survey on Video Diffusion Models](https://mlsanity.com/paper/2310.10647/) (2023)
- [Decomposing Motion and Content for Natural Video Sequence Prediction](https://mlsanity.com/paper/1706.08033/) (2017)
- [Neural SDEs as Infinite-Dimensional GANs](https://mlsanity.com/paper/2102.03657/) (2021)
- [TrajectoryNet: A Dynamic Optimal Transport Network for Modeling Cellular Dynamics](https://mlsanity.com/paper/2002.04461/) (2020)
- [Diffusion Models for Video Prediction and Infilling](https://mlsanity.com/paper/2206.07696/) (2022)
- [Deep Learning on Image Denoising: An overview](https://mlsanity.com/paper/1912.13171/) (2019)
- [Video Pixel Networks](https://mlsanity.com/paper/1610.00527/) (2016)
- [Simulation-free Schr"odinger bridges via score and flow matching](https://mlsanity.com/paper/2307.03672/) (2023)
- [Manifold Interpolating Optimal-Transport Flows for Trajectory Inference](https://mlsanity.com/paper/2206.14928/) (2022)
- [A Multimodal Transformer: Fusing Clinical Notes with Structured EHR Data for Interpretable In-Hospital Mortality Prediction](https://mlsanity.com/paper/2208.10240/) (2022)
- [From Similarity to Superiority: Channel Clustering for Time Series Forecasting](https://mlsanity.com/paper/2404.01340/) (2024)
- [Stable Neural Stochastic Differential Equations in Analyzing Irregular Time Series Data](https://mlsanity.com/paper/2402.14989/) (2024)

## Built on by (most cited first, 19 of 19)

- [Geometry-Aware Generative Autoencoders for Warped Riemannian Metric Learning and Generative Modeling on Data Manifolds](https://mlsanity.com/paper/2410.12779/) (2024)
- [ODE-GS: Latent ODEs for Dynamic Scene Extrapolation with 3D Gaussian Splatting](https://mlsanity.com/paper/2506.05480/) (2025)
- [Learning Patient-Specific Disease Dynamics with Latent Flow Matching for Longitudinal Imaging Generation](https://mlsanity.com/paper/2512.09185/) (2025)
- [Longitudinal Flow Matching for Trajectory Modeling](https://mlsanity.com/paper/2510.03569/) (2025)
- [Temporal Flow Matching for Learning Spatio-Temporal Trajectories in 4D Longitudinal Medical Imaging](https://mlsanity.com/paper/2508.21580/) (2025)
- [Frozen Forecasting: A Unified Evaluation](https://mlsanity.com/paper/2507.13942/) (2025)
- [Medical world models: representing medical states, modelling clinical dynamics and guiding intervention policies](https://mlsanity.com/paper/2606.16721/) (2026)
- [Uncertainty in a Single Pass: A Closed-Form Identity for One-Step Flow Matching](https://mlsanity.com/paper/2605.00941/) (2026)
- [Early Lung Cancer Diagnosis from Virtual Follow-up LDCT Generation via Correlational Autoencoder and Latent Flow Matching](https://mlsanity.com/paper/2511.18185/) (2025)
- [Dispersion Loss Counteracts Embedding Condensation and Improves Generalization in Small Language Models](https://mlsanity.com/paper/2602.00217/) (2026)
- [Forecasting Future Anatomies: Longitudinal Brain Mri-to-Mri Prediction](https://mlsanity.com/paper/2511.02558/) (2025)
- [CRONOS: Continuous Time Reconstruction for 4D Medical Longitudinal Series](https://mlsanity.com/paper/2512.16577/) (2025)
- [TransFair: Transferring Fairness from Ocular Disease Classification to Progression Prediction](https://mlsanity.com/paper/2412.00051/) (2024)
- [STAGED: A Multi-Agent Neural Network for Learning Cellular Interaction Dynamics](https://mlsanity.com/paper/2507.11660/) (2025)
- [NAMD: Virtual Follow-up Computed Tomography Synthesis via Nodule-Aligned Multimodal Diffusion Models for Early Lung Cancer Diagnosis](https://mlsanity.com/paper/2603.15932/) (2026)
- [Diagnostic-Guided Longitudinal Modeling for Forecasting Retinal Atrophy Progression](https://mlsanity.com/paper/2604.16955/) (2026)
- [Physics-Grounded Disentangled Flow Modeling for Brain Disease Progression Trajectory](https://mlsanity.com/paper/2606.28630/) (2026)
- [Modelling Geographic Atrophy Progression using Implicit Neural Representations](https://mlsanity.com/paper/2608.10807/) (2026)
- [Forecasting Intrathecal Tracer Enhancement from Pre-Contrast Brain MRI: Direct Regression versus Flow Matching](https://mlsanity.com/paper/2609.22377/) (2026)

## Abstract (arXiv)

Advances in medical imaging technologies have enabled the collection of longitudinal images, which involve repeated scanning of the same patients over time, to monitor disease progression. However, predictive modeling of such data remains challenging due to high dimensionality, irregular sampling, and data sparsity. To address these issues, we propose ImageFlowNet, a novel model designed to forecast disease trajectories from initial images while preserving spatial details. ImageFlowNet first learns multiscale joint representation spaces across patients and time points, then optimizes deterministic or stochastic flow fields within these spaces using a position-parameterized neural ODE/SDE framework. The model leverages a UNet architecture to create robust multiscale representations and mitigates data scarcity by combining knowledge from all patients. We provide theoretical insights that support our formulation of ODEs, and motivate our regularizations involving high-level visual features, latent space organization, and trajectory smoothness. We validate ImageFlowNet on three longitudinal medical image datasets depicting progression in geographic atrophy, multiple sclerosis, and glioblastoma, demonstrating its ability to effectively forecast disease progression and outperform existing methods. Our contributions include the development of ImageFlowNet, its theoretical underpinnings, and empirical validation on real-world datasets. The official implementation is available at https://github.com/KrishnaswamyLab/ImageFlowNet.