# Integrating Sequence and Image Modeling in Irregular Medical Time Series Through Self-Supervised Learning

arXiv: https://arxiv.org/abs/2502.06134 · HTML: https://mlsanity.com/paper/2502.06134/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 36 references that are in the map.
- First posted on arXiv 10 Feb 2025.

## Builds on (most cited first, 17 of 17)

- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Big Self-Supervised Models are Strong Semi-Supervised Learners](https://mlsanity.com/paper/2006.10029/) (2020)
- [Swin Transformer V2: Scaling Up Capacity and Resolution](https://mlsanity.com/paper/2111.09883/) (2021)
- [Recurrent Neural Networks for Multivariate Time Series with Missing Values](https://mlsanity.com/paper/1606.01865/) (2016)
- [Multitask learning and benchmarking with clinical time series data](https://mlsanity.com/paper/1703.07771/) (2017)
- [Doctor AI: Predicting Clinical Events via Recurrent Neural Networks](https://mlsanity.com/paper/1511.05942/) (2015)
- [Set Functions for Time Series](https://mlsanity.com/paper/1909.12064/) (2019)
- [CARD: Classification and Regression Diffusion Models](https://mlsanity.com/paper/2206.07275/) (2022)
- [Phased LSTM: Accelerating Recurrent Network Training for Long or Event-based Sequences](https://mlsanity.com/paper/1610.09513/) (2016)
- [Imaging Time-Series to Improve Classification and Imputation](https://mlsanity.com/paper/1506.00327/) (2015)
- [Time Series as Images: Vision Transformer for Irregularly Sampled Time Series](https://mlsanity.com/paper/2303.12799/) (2023)
- [Deep Learning for Multivariate Time Series Imputation: A Survey](https://mlsanity.com/paper/2402.04059/) (2024)
- [Warpformer: A Multi-scale Modeling Approach for Irregular Clinical Time Series](https://mlsanity.com/paper/2306.09368/) (2023)
- [Classification of Time-Series Images Using Deep Convolutional Neural Networks](https://mlsanity.com/paper/1710.00886/) (2017)
- [Sequential Multi-Dimensional Self-Supervised Learning for Clinical Time Series](https://mlsanity.com/paper/2307.10923/) (2023)
- [Visual Time Series Forecasting: An Image-driven Approach](https://mlsanity.com/paper/2011.09052/) (2020)
- [MedalCare-XL: 16,900 healthy and pathological 12 lead ECGs obtained through electrophysiological simulations](https://mlsanity.com/paper/2211.15997/) (2022)

## Built on by (most cited first, 1 of 1)

- [DBGL: Decay-aware Bipartite Graph Learning for Irregular Medical Time Series Classification](https://mlsanity.com/paper/2604.11842/) (2026)

## Abstract (arXiv)

Medical time series are often irregular and face significant missingness, posing challenges for data analysis and clinical decision-making. Existing methods typically adopt a single modeling perspective, either treating series data as sequences or transforming them into image representations for further classification. In this paper, we propose a joint learning framework that incorporates both sequence and image representations. We also design three self-supervised learning strategies to facilitate the fusion of sequence and image representations, capturing a more generalizable joint representation. The results indicate that our approach outperforms seven other state-of-the-art models in three representative real-world clinical datasets. We further validate our approach by simulating two major types of real-world missingness through leave-sensors-out and leave-samples-out techniques. The results demonstrate that our approach is more robust and significantly surpasses other baselines in terms of classification performance.