# Guiding Masked Representation Learning to Capture Spatio-Temporal Relationship of Electrocardiogram

arXiv: https://arxiv.org/abs/2402.09450 · HTML: https://mlsanity.com/paper/2402.09450/

## Key facts

- Cited by 84 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 53 references that are in the map.
- First posted on arXiv 2 Feb 2024.

## Builds on (most cited first, 25 of 25)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [SimMIM: A Simple Framework for Masked Image Modeling](https://mlsanity.com/paper/2111.09886/) (2021)
- [Self-supervised Visual Feature Learning with Deep Neural Networks: A Survey](https://mlsanity.com/paper/1902.06162/) (2019)
- [CLOCS: Contrastive Learning of Cardiac Signals Across Space, Time, and Patients](https://mlsanity.com/paper/2005.13249/) (2020)
- [Automatic diagnosis of the 12-lead ECG using a deep neural network](https://mlsanity.com/paper/1904.01949/) (2019)
- [Deep Learning for ECG Analysis: Benchmarks and Insights from PTB-XL](https://mlsanity.com/paper/2004.13701/) (2020)
- [Self-supervised ECG Representation Learning for Emotion Recognition](https://mlsanity.com/paper/2002.03898/) (2020)
- [What Do Self-Supervised Vision Transformers Learn?](https://mlsanity.com/paper/2305.00729/) (2023)
- [Self-supervised representation learning from 12-lead ECG data](https://mlsanity.com/paper/2103.12676/) (2021)
- [Lead-agnostic Self-supervised Learning for Local and Global Representations of Electrocardiogram](https://mlsanity.com/paper/2203.06889/) (2022)
- [3KG: Contrastive Learning of 12-Lead Electrocardiograms using Physiologically-Inspired Augmentations](https://mlsanity.com/paper/2106.04452/) (2021)
- [Intra-Inter Subject Self-supervised Learning for Multivariate Cardiac Signals](https://mlsanity.com/paper/2109.08908/) (2021)
- [ECG arrhythmia classification using a 2-D convolutional neural network](https://mlsanity.com/paper/1804.06812/) (2018)
- [Analysis of Augmentations for Contrastive ECG Representation Learning](https://mlsanity.com/paper/2206.07656/) (2022)
- [MVKT-ECG: Efficient Single-lead ECG Classification on Multi-Label Arrhythmia by Multi-View Knowledge Transferring](https://mlsanity.com/paper/2301.12178/) (2023)
- [Data Augmentation for Electrocardiogram Classification with Deep Neural Network](https://mlsanity.com/paper/2009.04398/) (2020)

## Built on by (most cited first, 30 of 84)

- [ECG-FM: An Open Electrocardiogram Foundation Model](https://mlsanity.com/paper/2408.05178/) (2024)
- [Zero-Shot ECG Classification with Multimodal Learning and Test-time Clinical Knowledge Enhancement](https://mlsanity.com/paper/2403.06659/) (2024)
- [Reading Your Heart: Learning ECG Words and Sentences via Pre-training ECG Language Model](https://mlsanity.com/paper/2502.10707/) (2025)
- [MEIT: Multimodal Electrocardiogram Instruction Tuning on Large Language Models for Report Generation](https://mlsanity.com/paper/2403.04945/) (2024)
- [Teach Multimodal LLMs to Comprehend Electrocardiographic Images](https://mlsanity.com/paper/2410.19008/) (2024)
- [An Electrocardiogram Foundation Model Built on over 10 Million Recordings with External Evaluation across Multiple Domains](https://mlsanity.com/paper/2410.04133/) (2024)
- [LSM-2: Learning from Incomplete Wearable Sensor Data](https://mlsanity.com/paper/2506.05321/) (2025)
- [Learning General Representation of 12-Lead Electrocardiogram with a Joint-Embedding Predictive Architecture](https://mlsanity.com/paper/2410.08559/) (2024)
- [Boosting Masked ECG-Text Auto-Encoders as Discriminative Learners](https://mlsanity.com/paper/2410.02131/) (2024)
- [From Token to Rhythm: A Multi-Scale Approach for ECG-Language Pretraining](https://mlsanity.com/paper/2506.21803/) (2025)
- [Self-Supervised Pre-Training with Joint-Embedding Predictive Architecture Boosts ECG Classification Performance](https://mlsanity.com/paper/2410.13867/) (2024)
- [OSF: On Pre-training and Scaling of Sleep Foundation Models](https://mlsanity.com/paper/2603.00190/) (2026)
- [anyECG-chat: A Generalist ECG-MLLM for Flexible ECG Input and Multi-Task Understanding](https://mlsanity.com/paper/2506.00942/) (2025)
- [ECHOPulse: ECG controlled echocardio-grams video generation](https://mlsanity.com/paper/2410.03143/) (2024)
- [ECG-Byte: A Tokenizer for End-to-End Generative Electrocardiogram Language Modeling](https://mlsanity.com/paper/2412.14373/) (2024)
- [BenchECG and xECG: a benchmark and baseline for ECG foundation models](https://mlsanity.com/paper/2509.10151/) (2025)
- [ECG-R1: Protocol-Guided and Modality-Agnostic MLLM for Reliable ECG Interpretation](https://mlsanity.com/paper/2602.04279/) (2026)
- [Benchmarking ECG FMs: A Reality Check Across Clinical Tasks](https://mlsanity.com/paper/2509.25095/) (2025)
- [CLEF: Clinically-Guided Contrastive Learning for Electrocardiogram Foundation Models](https://mlsanity.com/paper/2512.02180/) (2025)
- [SuPreME: A Supervised Pre-training Framework for Multimodal ECG Representation Learning](https://mlsanity.com/paper/2502.19668/) (2025)
- [Global and Local Contrastive Learning for Joint Representations from Cardiac MRI and ECG](https://mlsanity.com/paper/2506.20683/) (2025)
- [AnyECG: Foundational Models for Multitask Cardiac Analysis in Real-World Settings](https://mlsanity.com/paper/2411.17711/) (2024)
- [Knowledge-enhanced Multimodal ECG Representation Learning with Arbitrary-Lead Inputs](https://mlsanity.com/paper/2502.17900/) (2025)
- [Self-Alignment Learning to Improve Myocardial Infarction Detection from Single-Lead ECG](https://mlsanity.com/paper/2509.19397/) (2025)
- [Translating Electrocardiograms to Cardiac Magnetic Resonance Imaging Useful for Cardiac Assessment and Disease Screening: A Multi-Center Study](https://mlsanity.com/paper/2411.13602/) (2024)
- [Signal, Image, or Symbolic: Exploring the Best Input Representation for Electrocardiogram-Language Models Through a Unified Framework](https://mlsanity.com/paper/2505.18847/) (2025)
- [Interpretable Multimodal Zero-Shot ECG Diagnosis via Structured Clinical Knowledge Alignment](https://mlsanity.com/paper/2510.21551/) (2025)
- [RhythmBERT: A Self-Supervised Language Model Based on Latent Representations of ECG Waveforms for Heart Disease Detection](https://mlsanity.com/paper/2602.23060/) (2026)
- [CardioState-JEPA: Delay-Aware Cross-Modal Learning of a Shared Cardiac Representation](https://mlsanity.com/paper/2608.12944/) (2026)
- [FOCAL: Fine-Grained Optimal-Transport-Driven Contrastive Alignment of Language and ECGs with Waveform Enhancement](https://mlsanity.com/paper/2505.11939/) (2025)

## Abstract (arXiv)

Electrocardiograms (ECG) are widely employed as a diagnostic tool for monitoring electrical signals originating from a heart. Recent machine learning research efforts have focused on the application of screening various diseases using ECG signals. However, adapting to the application of screening disease is challenging in that labeled ECG data are limited. Achieving general representation through self-supervised learning (SSL) is a well-known approach to overcome the scarcity of labeled data; however, a naive application of SSL to ECG data, without considering the spatial-temporal relationships inherent in ECG signals, may yield suboptimal results. In this paper, we introduce ST-MEM (Spatio-Temporal Masked Electrocardiogram Modeling), designed to learn spatio-temporal features by reconstructing masked 12-lead ECG data. ST-MEM outperforms other SSL baseline methods in various experimental settings for arrhythmia classification tasks. Moreover, we demonstrate that ST-MEM is adaptable to various lead combinations. Through quantitative and qualitative analysis, we show a spatio-temporal relationship within ECG data. Our code is available at https://github.com/bakqui/ST-MEM.