# Advancing Precision Oncology Through Modeling of Longitudinal and Multimodal Data

arXiv: https://arxiv.org/abs/2502.07836 · HTML: https://mlsanity.com/paper/2502.07836/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 184 references that are in the map.
- First posted on arXiv 11 Feb 2025.

## Builds on (most cited first, 30 of 37)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling](https://mlsanity.com/paper/1803.01271/) (2018)
- [Med-BERT: pre-trained contextualized embeddings on large-scale structured electronic health records for disease prediction](https://mlsanity.com/paper/2005.12833/) (2020)
- [Balanced Multimodal Learning via On-the-fly Gradient Modulation](https://mlsanity.com/paper/2203.15332/) (2022)
- [BEHRT: Transformer for Electronic Health Records](https://mlsanity.com/paper/1907.09538/) (2019)
- [Doctor AI: Predicting Clinical Events via Recurrent Neural Networks](https://mlsanity.com/paper/1511.05942/) (2015)
- [Large Language Models are Few-Shot Health Learners](https://mlsanity.com/paper/2305.15525/) (2023)
- [Dipole: Diagnosis Prediction in Healthcare via Attention-based Bidirectional Recurrent Neural Networks](https://mlsanity.com/paper/1706.05764/) (2017)
- [PMR: Prototypical Modal Rebalance for Multimodal Learning](https://mlsanity.com/paper/2211.07089/) (2022)
- [Large Models for Time Series and Spatio-Temporal Data: A Survey and Outlook](https://mlsanity.com/paper/2310.10196/) (2023)
- [Learning to Diagnose with LSTM Recurrent Neural Networks](https://mlsanity.com/paper/1511.03677/) (2015)
- [Quantifying & Modeling Multimodal Interactions: An Information Decomposition Framework](https://mlsanity.com/paper/2302.12247/) (2023)
- [Foundation Model for Advancing Healthcare: Challenges, Opportunities, and Future Directions](https://mlsanity.com/paper/2404.03264/) (2024)
- [Deep Reinforcement Learning for Sepsis Treatment](https://mlsanity.com/paper/1711.09602/) (2017)
- [RetainVis: Visual Analytics with Interpretable and Interactive Recurrent Neural Networks on Electronic Medical Records](https://mlsanity.com/paper/1805.10724/) (2018)
- [Hi-BEHRT: Hierarchical Transformer-based model for accurate prediction of clinical events using multimodal longitudinal electronic health records](https://mlsanity.com/paper/2106.11360/) (2021)
- [Learning to Detect Sepsis with a Multitask Gaussian Process RNN Classifier](https://mlsanity.com/paper/1706.04152/) (2017)
- [Federated Learning for Medical Applications: A Taxonomy, Current Trends, Challenges, and Future Research Directions](https://mlsanity.com/paper/2208.03392/) (2022)
- [Deep learning for temporal data representation in electronic health records: A systematic review of challenges and methodologies](https://mlsanity.com/paper/2107.09951/) (2021)
- [A scalable end-to-end Gaussian process adapter for irregularly sampled time series classification](https://mlsanity.com/paper/1606.04443/) (2016)
- [Self-Supervised Longitudinal Neighbourhood Embedding](https://mlsanity.com/paper/2103.03840/) (2021)
- [SurvLatent ODE : A Neural ODE based time-to-event model with competing risks for longitudinal data improves cancer-associated Venous Thromboembolism (VTE) prediction](https://mlsanity.com/paper/2204.09633/) (2022)
- [TINC: Temporally Informed Non-Contrastive Learning for Disease Progression Modeling in Retinal OCT Volumes](https://mlsanity.com/paper/2206.15282/) (2022)
- [Learning to Compare Longitudinal Images](https://mlsanity.com/paper/2304.02531/) (2023)
- [DoubleMLDeep: Estimation of Causal Effects with Multimodal Data](https://mlsanity.com/paper/2402.01785/) (2024)
- [Medical Time Series Classification with Hierarchical Attention-based Temporal Convolutional Networks: A Case Study of Myotonic Dystrophy Diagnosis](https://mlsanity.com/paper/1903.11748/) (2019)
- [Temporal Supervised Contrastive Learning for Modeling Patient Risk Progression](https://mlsanity.com/paper/2312.05933/) (2023)

## Built on by (most cited first, 3 of 3)

- [Multimodal Disease Progression Modeling via Spatiotemporal Disentanglement and Multiscale Alignment](https://mlsanity.com/paper/2510.11112/) (2025)
- [TwinWeaver: An LLM-Based Foundation Model Framework for Pan-Cancer Digital Twins](https://mlsanity.com/paper/2601.20906/) (2026)
- [Routine laboratory trajectories encode the onset of organ-level complications in cancer](https://mlsanity.com/paper/2606.08538/) (2026)

## Abstract (arXiv)

Cancer evolves continuously over time through a complex interplay of genetic, epigenetic, microenvironmental, and phenotypic changes. This dynamic behavior drives uncontrolled cell growth, metastasis, immune evasion, and therapy resistance, posing challenges for effective monitoring and treatment. However, today's data-driven research in oncology has primarily focused on cross-sectional analysis using data from a single modality, limiting the ability to fully characterize and interpret the disease's dynamic heterogeneity. Advances in multiscale data collection and computational methods now enable the discovery of longitudinal multimodal biomarkers for precision oncology. Longitudinal data reveal patterns of disease progression and treatment response that are not evident from single-timepoint data, enabling timely abnormality detection and dynamic treatment adaptation. Multimodal data integration offers complementary information from diverse sources for more precise risk assessment and targeting of cancer therapy. In this review, we survey methods of longitudinal and multimodal modeling, highlighting their synergy in providing multifaceted insights for personalized care tailored to the unique characteristics of a patient's cancer. We summarize the current challenges and future directions of longitudinal multimodal analysis in advancing precision oncology.