# Self-supervised representation learning from 12-lead ECG data

arXiv: https://arxiv.org/abs/2103.12676 · HTML: https://mlsanity.com/paper/2103.12676/

## Key facts

- Cited by 80 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 40 references that are in the map.
- First posted on arXiv 23 Mar 2021.

## Builds on (most cited first, 22 of 22)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Bag of Tricks for Image Classification with Convolutional Neural Networks](https://mlsanity.com/paper/1812.01187/) (2018)
- [Big Self-Supervised Models Advance Medical Image Classification](https://mlsanity.com/paper/2101.05224/) (2021)
- [An Unsupervised Autoregressive Model for Speech Representation Learning](https://mlsanity.com/paper/1904.03240/) (2019)
- [CLOCS: Contrastive Learning of Cardiac Signals Across Space, Time, and Patients](https://mlsanity.com/paper/2005.13249/) (2020)
- [Automatic diagnosis of the 12-lead ECG using a deep neural network](https://mlsanity.com/paper/1904.01949/) (2019)
- [Deep Learning for ECG Analysis: Benchmarks and Insights from PTB-XL](https://mlsanity.com/paper/2004.13701/) (2020)
- [Uncovering the structure of clinical EEG signals with self-supervised learning](https://mlsanity.com/paper/2007.16104/) (2020)
- [Self-supervised ECG Representation Learning for Emotion Recognition](https://mlsanity.com/paper/2002.03898/) (2020)
- [Subject-Aware Contrastive Learning for Biosignals](https://mlsanity.com/paper/2007.04871/) (2020)
- [A Framework For Contrastive Self-Supervised Learning And Designing A New Approach](https://mlsanity.com/paper/2009.00104/) (2020)
- [Self-supervised representation learning from electroencephalography signals](https://mlsanity.com/paper/1911.05419/) (2019)
- [Opportunities and Challenges of Deep Learning Methods for Electrocardiogram Data: A Systematic Review](https://mlsanity.com/paper/2001.01550/) (2019)
- [A comparison of self-supervised speech representations as input features for unsupervised acoustic word embeddings](https://mlsanity.com/paper/2012.07387/) (2020)
- [Analysis of Predictive Coding Models for Phonemic Representation Learning in Small Datasets](https://mlsanity.com/paper/2007.04205/) (2020)

## Built on by (most cited first, 30 of 80)

- [Large Models for Time Series and Spatio-Temporal Data: A Survey and Outlook](https://mlsanity.com/paper/2310.10196/) (2023)
- [Large-scale Training of Foundation Models for Wearable Biosignals](https://mlsanity.com/paper/2312.05409/) (2023)
- [Guiding Masked Representation Learning to Capture Spatio-Temporal Relationship of Electrocardiogram](https://mlsanity.com/paper/2402.09450/) (2024)
- [Frozen Language Model Helps ECG Zero-Shot Learning](https://mlsanity.com/paper/2303.12311/) (2023)
- [Lead-agnostic Self-supervised Learning for Local and Global Representations of Electrocardiogram](https://mlsanity.com/paper/2203.06889/) (2022)
- [3KG: Contrastive Learning of 12-Lead Electrocardiograms using Physiologically-Inspired Augmentations](https://mlsanity.com/paper/2106.04452/) (2021)
- [Intra-Inter Subject Self-supervised Learning for Multivariate Cardiac Signals](https://mlsanity.com/paper/2109.08908/) (2021)
- [Toward Foundation Model for Multivariate Wearable Sensing of Physiological Signals](https://mlsanity.com/paper/2412.09758/) (2024)
- [ETP: Learning Transferable ECG Representations via ECG-Text Pre-training](https://mlsanity.com/paper/2309.07145/) (2023)
- [ECG Semantic Integrator (ESI): A Foundation ECG Model Pretrained with LLM-Enhanced Cardiological Text](https://mlsanity.com/paper/2405.19366/) (2024)
- [Learning General Representation of 12-Lead Electrocardiogram with a Joint-Embedding Predictive Architecture](https://mlsanity.com/paper/2410.08559/) (2024)
- [Wearable Accelerometer Foundation Models for Health via Knowledge Distillation](https://mlsanity.com/paper/2412.11276/) (2024)
- [Explaining Deep Learning for ECG Analysis: Building Blocks for Auditing and Knowledge Discovery](https://mlsanity.com/paper/2305.17043/) (2023)
- [Towards quantitative precision for ECG analysis: Leveraging state space models, self-supervision and patient metadata](https://mlsanity.com/paper/2308.15291/) (2023)
- [Towards Domain Generalization for ECG and EEG Classification: Algorithms and Benchmarks](https://mlsanity.com/paper/2303.11338/) (2023)
- [Self-Supervised Pre-Training with Joint-Embedding Predictive Architecture Boosts ECG Classification Performance](https://mlsanity.com/paper/2410.13867/) (2024)
- [Data Augmentation for Electrocardiograms](https://mlsanity.com/paper/2204.04360/) (2022)
- [From Modern CNNs to Vision Transformers: Assessing the Performance, Robustness, and Classification Strategies of Deep Learning Models in Histopathology](https://mlsanity.com/paper/2204.05044/) (2022)
- [Advancing the State-of-the-Art for ECG Analysis through Structured State Space Models](https://mlsanity.com/paper/2211.07579/) (2022)
- [Deep learning based ECG segmentation for delineation of diverse arrhythmias](https://mlsanity.com/paper/2304.06237/) (2023)
- [In-Distribution and Out-of-Distribution Self-supervised ECG Representation Learning for Arrhythmia Detection](https://mlsanity.com/paper/2304.06427/) (2023)
- [Masked Transformer for Electrocardiogram Classification](https://mlsanity.com/paper/2309.07136/) (2023)
- [S4Sleep: Elucidating the design space of deep-learning-based sleep stage classification models](https://mlsanity.com/paper/2310.06715/) (2023)
- [Using explainable AI to investigate electrocardiogram changes during healthy aging -- from expert features to raw signals](https://mlsanity.com/paper/2310.07463/) (2023)
- [CLEF: Clinically-Guided Contrastive Learning for Electrocardiogram Foundation Models](https://mlsanity.com/paper/2512.02180/) (2025)
- [MedalCare-XL: 16,900 healthy and pathological 12 lead ECGs obtained through electrophysiological simulations](https://mlsanity.com/paper/2211.15997/) (2022)
- [A Domain Generalization Approach for Out-Of-Distribution 12-lead ECG Classification with Convolutional Neural Networks](https://mlsanity.com/paper/2208.09656/) (2022)
- [Automated Contrastive Learning Strategy Search for Time Series](https://mlsanity.com/paper/2403.12641/) (2024)
- [Guidelines for Augmentation Selection in Contrastive Learning for Time Series Classification](https://mlsanity.com/paper/2407.09336/) (2024)
- [AnyECG: Foundational Models for Multitask Cardiac Analysis in Real-World Settings](https://mlsanity.com/paper/2411.17711/) (2024)

## Abstract (arXiv)

Clinical 12-lead electrocardiography (ECG) is one of the most widely encountered kinds of biosignals. Despite the increased availability of public ECG datasets, label scarcity remains a central challenge in the field. Self-supervised learning represents a promising way to alleviate this issue. In this work, we put forward the first comprehensive assessment of self-supervised representation learning from clinical 12-lead ECG data. To this end, we adapt state-of-the-art self-supervised methods based on instance discrimination and latent forecasting to the ECG domain. In a first step, we learn contrastive representations and evaluate their quality based on linear evaluation performance on a recently established, comprehensive, clinical ECG classification task. In a second step, we analyze the impact of self-supervised pretraining on finetuned ECG classifiers as compared to purely supervised performance. For the best-performing method, an adaptation of contrastive predictive coding, we find a linear evaluation performance only 0.5% below supervised performance. For the finetuned models, we find improvements in downstream performance of roughly 1% compared to supervised performance, label efficiency, as well as robustness against physiological noise. This work clearly establishes the feasibility of extracting discriminative representations from ECG data via self-supervised learning and the numerous advantages when finetuning such representations on downstream tasks as compared to purely supervised training. As first comprehensive assessment of its kind in the ECG domain carried out exclusively on publicly available datasets, we hope to establish a first step towards reproducible progress in the rapidly evolving field of representation learning for biosignals.