# Sequential Multi-Dimensional Self-Supervised Learning for Clinical Time Series

arXiv: https://arxiv.org/abs/2307.10923 · HTML: https://mlsanity.com/paper/2307.10923/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 48 references that are in the map.
- First posted on arXiv 20 Jul 2023.

## Builds on (most cited first, 30 of 30)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [Big Self-Supervised Models are Strong Semi-Supervised Learners](https://mlsanity.com/paper/2006.10029/) (2020)
- [VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning](https://mlsanity.com/paper/2105.04906/) (2021)
- [Multitask learning and benchmarking with clinical time series data](https://mlsanity.com/paper/1703.07771/) (2017)
- [Self-Supervised Contrastive Pre-Training For Time Series via Time-Frequency Consistency](https://mlsanity.com/paper/2206.08496/) (2022)
- [Unsupervised Representation Learning for Time Series with Temporal Neighborhood Coding](https://mlsanity.com/paper/2106.00750/) (2021)
- [CLOCS: Contrastive Learning of Cardiac Signals Across Space, Time, and Patients](https://mlsanity.com/paper/2005.13249/) (2020)
- [SCARF: Self-Supervised Contrastive Learning using Random Feature Corruption](https://mlsanity.com/paper/2106.15147/) (2021)
- [An Empirical Survey of Data Augmentation for Time Series Classification with Neural Networks](https://mlsanity.com/paper/2007.15951/) (2020)
- [VICRegL: Self-Supervised Learning of Local Visual Features](https://mlsanity.com/paper/2210.01571/) (2022)
- [Self-Supervised Transformer for Sparse and Irregularly Sampled Multivariate Clinical Time-Series](https://mlsanity.com/paper/2107.14293/) (2021)
- [Subject-Aware Contrastive Learning for Biosignals](https://mlsanity.com/paper/2007.04871/) (2020)
- [Hi-BEHRT: Hierarchical Transformer-based model for accurate prediction of clinical events using multimodal longitudinal electronic health records](https://mlsanity.com/paper/2106.11360/) (2021)
- [MedAug: Contrastive learning leveraging patient metadata improves representations for chest X-ray interpretation](https://mlsanity.com/paper/2102.10663/) (2021)
- [Transfer Learning with Deep Tabular Models](https://mlsanity.com/paper/2206.15306/) (2022)
- [Lead-agnostic Self-supervised Learning for Local and Global Representations of Electrocardiogram](https://mlsanity.com/paper/2203.06889/) (2022)
- [3KG: Contrastive Learning of 12-Lead Electrocardiograms using Physiologically-Inspired Augmentations](https://mlsanity.com/paper/2106.04452/) (2021)
- [Neighborhood Contrastive Learning Applied to Online Patient Monitoring](https://mlsanity.com/paper/2106.05142/) (2021)
- [Meta-Learning to Improve Pre-Training](https://mlsanity.com/paper/2111.01754/) (2021)
- [Local Spatiotemporal Representation Learning for Longitudinally-consistent Neuroimage Analysis](https://mlsanity.com/paper/2206.04281/) (2022)
- [Decoupling Local and Global Representations of Time Series](https://mlsanity.com/paper/2202.02262/) (2022)
- [Data Augmentation for Electrocardiograms](https://mlsanity.com/paper/2204.04360/) (2022)

## Built on by (most cited first, 16 of 16)

- [Scaling Wearable Foundation Models](https://mlsanity.com/paper/2410.13638/) (2024)
- [Event-Based Contrastive Learning for Medical Time Series](https://mlsanity.com/paper/2312.10308/) (2023)
- [Temporal Supervised Contrastive Learning for Modeling Patient Risk Progression](https://mlsanity.com/paper/2312.05933/) (2023)
- [UniCL: A Universal Contrastive Learning Framework for Large Time Series Models](https://mlsanity.com/paper/2405.10597/) (2024)
- [Deep Metric Learning for the Hemodynamics Inference with Electrocardiogram Signals](https://mlsanity.com/paper/2308.04650/) (2023)
- [Investigating a domain adaptation approach for integrating different measurement instruments in a longitudinal clinical registry](https://mlsanity.com/paper/2312.00616/) (2023)
- [Recent Advances, Applications, and Open Challenges in Machine Learning for Health: Reflections from Research Roundtables at ML4H 2023 Symposium](https://mlsanity.com/paper/2403.01628/) (2024)
- [Neural Fourier Modelling: A Highly Compact Approach to Time-Series Analysis](https://mlsanity.com/paper/2410.04703/) (2024)
- [Wearable Foundation Models Should Go Beyond Static Encoders](https://mlsanity.com/paper/2603.19564/) (2026)
- [A Unified Framework for Modeling Heterogeneous Financial Data via Dual-Granularity Prompting](https://mlsanity.com/paper/2404.13004/) (2024)
- [OC-Distill: Ontology-aware Contrastive Learning with Cross-Modal Distillation for ICU Risk Prediction](https://mlsanity.com/paper/2604.16878/) (2026)
- [Integrating Sequence and Image Modeling in Irregular Medical Time Series Through Self-Supervised Learning](https://mlsanity.com/paper/2502.06134/) (2025)
- [Semi-Supervised Generative Models for Disease Trajectories: A Case Study on Systemic Sclerosis](https://mlsanity.com/paper/2407.11427/) (2024)
- [Towards Unified Approaches in Self-Supervised Event Stream Modeling: Progress and Prospects](https://mlsanity.com/paper/2502.04899/) (2025)
- [MATA-Former & SIICU: Semantic Aware Temporal Alignment for High-Fidelity ICU Risk Prediction](https://mlsanity.com/paper/2604.01727/) (2026)
- [SL-S4Wave: Self-Supervised Learning of Physiological Waveforms with Structured State Space Models](https://mlsanity.com/paper/2606.19888/) (2026)

## Abstract (arXiv)

Self-supervised learning (SSL) for clinical time series data has received significant attention in recent literature, since these data are highly rich and provide important information about a patient's physiological state. However, most existing SSL methods for clinical time series are limited in that they are designed for unimodal time series, such as a sequence of structured features (e.g., lab values and vitals signs) or an individual high-dimensional physiological signal (e.g., an electrocardiogram). These existing methods cannot be readily extended to model time series that exhibit multimodality, with structured features and high-dimensional data being recorded at each timestep in the sequence. In this work, we address this gap and propose a new SSL method -- Sequential Multi-Dimensional SSL -- where a SSL loss is applied both at the level of the entire sequence and at the level of the individual high-dimensional data points in the sequence in order to better capture information at both scales. Our strategy is agnostic to the specific form of loss function used at each level -- it can be contrastive, as in SimCLR, or non-contrastive, as in VICReg. We evaluate our method on two real-world clinical datasets, where the time series contains sequences of (1) high-frequency electrocardiograms and (2) structured data from lab values and vitals signs. Our experimental results indicate that pre-training with our method and then fine-tuning on downstream tasks improves performance over baselines on both datasets, and in several settings, can lead to improvements across different self-supervised loss functions.