# MultiSigBERT: Beyond Survival Analysis through Multimodal and Sequential Modeling in Oncology

arXiv: https://arxiv.org/abs/2608.16972 · HTML: https://mlsanity.com/paper/2608.16972/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 8 of its 25 references that are in the map.
- First posted on arXiv 17 Aug 2026.

## Builds on (most cited first, 8 of 8)

- [CamemBERT: a Tasty French Language Model](https://mlsanity.com/paper/1911.03894/) (2019)
- [Time-to-Event Prediction with Neural Networks and Cox Regression](https://mlsanity.com/paper/1907.00825/) (2019)
- [A Primer on the Signature Method in Machine Learning](https://mlsanity.com/paper/1603.03788/) (2016)
- [DySurv: dynamic deep learning model for survival analysis with conditional variational inference](https://mlsanity.com/paper/2310.18681/) (2023)
- [Dynamical Survival Analysis with Controlled Latent States](https://mlsanity.com/paper/2401.17077/) (2024)
- [Individual dynamic prediction of clinical endpoint from large dimensional longitudinal biomarker history: a landmark approach](https://mlsanity.com/paper/2102.01466/) (2021)
- [Survival Seq2Seq: A Survival Model based on Sequence to Sequence Architecture](https://mlsanity.com/paper/2204.04542/) (2022)
- [Penalized regression calibration: a method for the prediction of survival outcomes using complex longitudinal and high-dimensional data](https://mlsanity.com/paper/2101.04426/) (2021)

## Abstract (arXiv)

Machine learning has become an essential component of modern healthcare, where the integration of heterogeneous data sources offers unprecedented opportunities to improve clinical decision-making. Electronic Health Records (EHR) contain complementary information -- including narrative clinical reports, numerical measurements, and structured variables -- yet most survival models remain limited to a single modality or fail to exploit the temporal nature of patient trajectories. We propose MultiSigBERT, a unified framework for multimodal sequential survival modeling in oncology based on path signature representations. Here, narrative medical reports (free-text) are converted into sentence embeddings by extracting and averaging contextual word embeddings. These representations are then compressed via modality-specific PCA and concatenated with structured covariates to form joint temporal trajectories which are then encoded using the Signature transform, a tool from Rough Paths theory that efficiently captures higher-order temporal interactions across modalities without supervision needed. The computed Signature features are finally incorporated as high dimensional features into a LASSO-regularized Cox model to estimate individualized risk scores. The performance of our novel MultiSigBERT pipeline is illustrated on the analysis of a real-world oncology cohort from the L'eon B'erard Center, comprising over 120,000 medical reports and structured records from more than 2,500 patients. The model achieves a concordance index of 0.743 (sd 0.029) on an independent test set, demonstrating the benefit of jointly modeling multimodal temporal dynamics together with patient-level geometric structure for survival prediction.