# ECGBERT: Understanding Hidden Language of ECGs with Self-Supervised Representation Learning

arXiv: https://arxiv.org/abs/2306.06340 · HTML: https://mlsanity.com/paper/2306.06340/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 57 references that are in the map.
- First posted on arXiv 10 Jun 2023.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RT-1: Robotics Transformer for Real-World Control at Scale](https://mlsanity.com/paper/2212.06817/) (2022)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021)
- [Med-BERT: pre-trained contextualized embeddings on large-scale structured electronic health records for disease prediction](https://mlsanity.com/paper/2005.12833/) (2020)
- [BEHRT: Transformer for Electronic Health Records](https://mlsanity.com/paper/1907.09538/) (2019)
- [QASC: A Dataset for Question Answering via Sentence Composition](https://mlsanity.com/paper/1910.11473/) (2019)
- [GatorTron: A Large Clinical Language Model to Unlock Patient Information from Unstructured Electronic Health Records](https://mlsanity.com/paper/2203.03540/) (2022)
- [Deep Learning for ECG Analysis: Benchmarks and Insights from PTB-XL](https://mlsanity.com/paper/2004.13701/) (2020)
- [ECG Heartbeat Classification: A Deep Transferable Representation](https://mlsanity.com/paper/1805.00794/) (2018)
- [Controlled Text Generation as Continuous Optimization with Multiple Constraints](https://mlsanity.com/paper/2108.01850/) (2021)
- [Fine-Tuning Large Neural Language Models for Biomedical Natural Language Processing](https://mlsanity.com/paper/2112.07869/) (2021)
- [Cardiologist-Level Arrhythmia Detection with Convolutional Neural Networks](https://mlsanity.com/paper/1707.01836/) (2017)
- [Transfer Knowledge from Natural Language to Electrocardiography: Can We Detect Cardiovascular Disease Through Language Models?](https://mlsanity.com/paper/2301.09017/) (2023)
- [Inter- and intra- patient ECG heartbeat classification for arrhythmia detection: a sequence to sequence deep learning approach](https://mlsanity.com/paper/1812.07421/) (2018)
- [HAN-ECG: An Interpretable Atrial Fibrillation Detection Model Using Hierarchical Attention Networks](https://mlsanity.com/paper/2002.05262/) (2020)
- [Large Language Models as Fiduciaries: A Case Study Toward Robustly Communicating With Artificial Intelligence Through Legal Standards](https://mlsanity.com/paper/2301.10095/) (2023)
- [ECG Arrhythmia Classification Using Transfer Learning from 2-Dimensional Deep CNN Features](https://mlsanity.com/paper/1812.04693/) (2018)
- [ECG Language Processing (ELP): a New Technique to Analyze ECG Signals](https://mlsanity.com/paper/2006.08841/) (2020)

## Built on by (most cited first, 12 of 12)

- [ECG-FM: An Open Electrocardiogram Foundation Model](https://mlsanity.com/paper/2408.05178/) (2024)
- [Reading Your Heart: Learning ECG Words and Sentences via Pre-training ECG Language Model](https://mlsanity.com/paper/2502.10707/) (2025)
- [Enhancing Foundation Models for Time Series Forecasting via Wavelet-based Tokenization](https://mlsanity.com/paper/2412.05244/) (2024)
- [ECG-Byte: A Tokenizer for End-to-End Generative Electrocardiogram Language Modeling](https://mlsanity.com/paper/2412.14373/) (2024)
- [HeartBERT: A Self-Supervised ECG Embedding Model for Efficient and Effective Medical Signal Analysis](https://mlsanity.com/paper/2411.11896/) (2024)
- [Generative AI for Healthcare: Fundamentals, Challenges, and Perspectives](https://mlsanity.com/paper/2510.24551/) (2025)
- [RhythmBERT: A Self-Supervised Language Model Based on Latent Representations of ECG Waveforms for Heart Disease Detection](https://mlsanity.com/paper/2602.23060/) (2026)
- [LiPCoT: Linear Predictive Coding based Tokenizer for Self-supervised Learning of Time Series Data via Language Models](https://mlsanity.com/paper/2408.07292/) (2024)
- [Generalist vs Specialist Time Series Foundation Models: Investigating Potential Emergent Behaviors in Assessing Human Health Using PPG Signals](https://mlsanity.com/paper/2510.14254/) (2025)
- [Large language models in medical time series analysis](https://mlsanity.com/paper/2609.22262/) (2026)
- [LiteEMG-FM: An Efficient and Deployable Foundation Model for Robust EMG Sensing](https://mlsanity.com/paper/2610.02497/) (2026)
- [EMG-FM-Bench: A Comprehensive Benchmark for Foundation Model Transfer and Adaptation on Electromyography](https://mlsanity.com/paper/2610.06450/) (2026)

## Abstract (arXiv)

In the medical field, current ECG signal analysis approaches rely on supervised deep neural networks trained for specific tasks that require substantial amounts of labeled data. However, our paper introduces ECGBERT, a self-supervised representation learning approach that unlocks the underlying language of ECGs. By unsupervised pre-training of the model, we mitigate challenges posed by the lack of well-labeled and curated medical data. ECGBERT, inspired by advances in the area of natural language processing and large language models, can be fine-tuned with minimal additional layers for various ECG-based problems. Through four tasks, including Atrial Fibrillation arrhythmia detection, heartbeat classification, sleep apnea detection, and user authentication, we demonstrate ECGBERT's potential to achieve state-of-the-art results on a wide variety of tasks.