# HeartBERT: A Self-Supervised ECG Embedding Model for Efficient and Effective Medical Signal Analysis

arXiv: https://arxiv.org/abs/2411.11896 · HTML: https://mlsanity.com/paper/2411.11896/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 69 references that are in the map.
- First posted on arXiv 8 Nov 2024.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [An Introduction to Convolutional Neural Networks](https://mlsanity.com/paper/1511.08458/) (2015)
- [Self-supervised representation learning from 12-lead ECG data](https://mlsanity.com/paper/2103.12676/) (2021)
- [HAN-ECG: An Interpretable Atrial Fibrillation Detection Model Using Hierarchical Attention Networks](https://mlsanity.com/paper/2002.05262/) (2020)
- [Self-supervised Learning for Label-Efficient Sleep Stage Classification: A Comprehensive Evaluation](https://mlsanity.com/paper/2210.06286/) (2022)
- [Icentia11K: An Unsupervised Representation Learning Dataset for Arrhythmia Subtype Discovery](https://mlsanity.com/paper/1910.09570/) (2019)
- [ECGBERT: Understanding Hidden Language of ECGs with Self-Supervised Representation Learning](https://mlsanity.com/paper/2306.06340/) (2023)
- [ECG Heartbeat Classification Using Multimodal Fusion](https://mlsanity.com/paper/2107.09869/) (2021)
- [Large Language Model-informed ECG Dual Attention Network for Heart Failure Risk Prediction](https://mlsanity.com/paper/2403.10581/) (2024)
- [ECG Language Processing (ELP): a New Technique to Analyze ECG Signals](https://mlsanity.com/paper/2006.08841/) (2020)
- [In-Distribution and Out-of-Distribution Self-supervised ECG Representation Learning for Arrhythmia Detection](https://mlsanity.com/paper/2304.06427/) (2023)
- [Unsupervised Pre-Training Using Masked Autoencoders for ECG Analysis](https://mlsanity.com/paper/2310.11153/) (2023)
- [ECGformer: Leveraging transformer for ECG heartbeat arrhythmia classification](https://mlsanity.com/paper/2401.05434/) (2024)
- [The Effect of Data Augmentation on Classification of Atrial Fibrillation in Short Single-Lead ECG Signals Using Deep Neural Networks](https://mlsanity.com/paper/2002.02870/) (2020)
- [Generative Pre-Trained Transformer for Cardiac Abnormality Detection](https://mlsanity.com/paper/2110.04071/) (2021)

## Built on by (most cited first, 3 of 3)

- [ECG-Byte: A Tokenizer for End-to-End Generative Electrocardiogram Language Modeling](https://mlsanity.com/paper/2412.14373/) (2024)
- [The Impact of Temporal Context Length and Encoding Strategies on Self-Supervised ECG Representation Learning](https://mlsanity.com/paper/2608.12695/) (2026)
- [Retrieval-Augmented Multimodal Language Model with Heterogeneous Evidence Integration for Clinical Prognosis](https://mlsanity.com/paper/2610.13510/) (2026)

## Abstract (arXiv)

The HeartBert model is introduced with three primary objectives: reducing the need for labeled data, minimizing computational resources, and simultaneously improving performance in machine learning systems that analyze Electrocardiogram (ECG) signals. Inspired by Bidirectional Encoder Representations from Transformers (BERT) in natural language processing and enhanced with a self-supervised learning approach, the HeartBert model-built on the RoBERTa architecture-generates sophisticated embeddings tailored for ECG-based projects in the medical domain. To demonstrate the versatility, generalizability, and efficiency of the proposed model, two key downstream tasks have been selected: sleep stage detection and heartbeat classification. HeartBERT-based systems, utilizing bidirectional LSTM heads, are designed to address complex challenges. A series of practical experiments have been conducted to demonstrate the superiority and advancements of HeartBERT, particularly in terms of its ability to perform well with smaller training datasets, reduced learning parameters, and effective performance compared to rival models. The code and data are publicly available at https://github.com/ecgResearch/HeartBert.