# From Token to Rhythm: A Multi-Scale Approach for ECG-Language Pretraining

arXiv: https://arxiv.org/abs/2506.21803 · HTML: https://mlsanity.com/paper/2506.21803/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 48 references that are in the map.
- First posted on arXiv 11 Jun 2025.

## Builds on (most cited first, 30 of 31)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [A Time Series is Worth 64 Words: Long-term Forecasting with Transformers](https://mlsanity.com/paper/2211.14730/) (2022)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [CoCa: Contrastive Captioners are Image-Text Foundation Models](https://mlsanity.com/paper/2205.01917/) (2022)
- [Unified Language Model Pre-training for Natural Language Understanding and Generation](https://mlsanity.com/paper/1905.03197/) (2019)
- [TS2Vec: Towards Universal Representation of Time Series](https://mlsanity.com/paper/2106.10466/) (2021)
- [Time-Series Representation Learning via Temporal and Contextual Contrasting](https://mlsanity.com/paper/2106.14112/) (2021)
- [Making the Most of Text Semantics to Improve Biomedical Vision--Language Processing](https://mlsanity.com/paper/2204.09817/) (2022)
- [MedCPT: Contrastive Pre-trained Transformers with Large-scale PubMed Search Logs for Zero-shot Biomedical Information Retrieval](https://mlsanity.com/paper/2307.00589/) (2023)
- [CLOCS: Contrastive Learning of Cardiac Signals Across Space, Time, and Patients](https://mlsanity.com/paper/2005.13249/) (2020)
- [ECG-FM: An Open Electrocardiogram Foundation Model](https://mlsanity.com/paper/2408.05178/) (2024)
- [Guiding Masked Representation Learning to Capture Spatio-Temporal Relationship of Electrocardiogram](https://mlsanity.com/paper/2402.09450/) (2024)
- [Zero-Shot ECG Classification with Multimodal Learning and Test-time Clinical Knowledge Enhancement](https://mlsanity.com/paper/2403.06659/) (2024)
- [Frozen Language Model Helps ECG Zero-Shot Learning](https://mlsanity.com/paper/2303.12311/) (2023)
- [Self-Supervised Time Series Representation Learning via Cross Reconstruction Transformer](https://mlsanity.com/paper/2205.09928/) (2022)
- [Lead-agnostic Self-supervised Learning for Local and Global Representations of Electrocardiogram](https://mlsanity.com/paper/2203.06889/) (2022)
- [ECG-Chat: A Large ECG-Language Model for Cardiac Disease Diagnosis](https://mlsanity.com/paper/2408.08849/) (2024)
- [Reading Your Heart: Learning ECG Words and Sentences via Pre-training ECG Language Model](https://mlsanity.com/paper/2502.10707/) (2025)
- [Teach Multimodal LLMs to Comprehend Electrocardiographic Images](https://mlsanity.com/paper/2410.19008/) (2024)
- [ECG Semantic Integrator (ESI): A Foundation ECG Model Pretrained with LLM-Enhanced Cardiological Text](https://mlsanity.com/paper/2405.19366/) (2024)
- [CREMA: A Contrastive Regularized Masked Autoencoder for Robust ECG Diagnostics across Clinical Domains](https://mlsanity.com/paper/2407.07110/) (2024)
- [A Systematic Review on Foundation Models for Electrocardiogram Analysis: Initial Strides and Expansive Horizons](https://mlsanity.com/paper/2410.19877/) (2024)

## Built on by (most cited first, 16 of 16)

- [Time Series, Vision, and Language: Exploring the Limits of Alignment in Contrastive Representation Spaces](https://mlsanity.com/paper/2602.19367/) (2026)
- [Signal or Noise? Understanding Generative Models for Real-World Sensor Time Series](https://mlsanity.com/paper/2607.04245/) (2026)
- [LVCG: Learning ECG Representations in the Latent Vectorcardiogram Space](https://mlsanity.com/paper/2605.31249/) (2026)
- [CAMEL: An ECG Language Model for Forecasting Cardiac Events](https://mlsanity.com/paper/2602.15677/) (2026)
- [Position: Evaluation of ECG Representations Must Be Fixed](https://mlsanity.com/paper/2602.17531/) (2026)
- [Learning ECG Image Representations via Dual Physiological-Aware Alignments](https://mlsanity.com/paper/2604.01526/) (2026)
- [PanLUNA: An Efficient and Robust Query-Unified Multimodal Model for Edge Biosignal Intelligence](https://mlsanity.com/paper/2604.04297/) (2026)
- [PRISM-CTG: A Foundation Model for Cardiotocography Analysis with Multi-View SSL](https://mlsanity.com/paper/2605.02917/) (2026)
- [Reasoning Before Diagnosis: Physician-Inspired Structured Thinking for ECG Classification](https://mlsanity.com/paper/2605.17308/) (2026)
- [Information-theoretic Multimodal Representation Learning for Electrocardiogram Signals](https://mlsanity.com/paper/2605.27583/) (2026)
- [Failures Reveal What Metrics Miss: An Evidence-Driven Agent for Recursive Refinement of ECG Classifiers](https://mlsanity.com/paper/2607.24419/) (2026)
- [Beat-Synchronous Tokenization for ECG Transformers](https://mlsanity.com/paper/2608.30367/) (2026)
- [SHIFT-M3: Pre-fusion Alignment-based Consistency Screening for Multimodal ECG Record Integrity](https://mlsanity.com/paper/2609.13874/) (2026)
- [NeuroECG: ECGFounder-Based Deep ECG Representation for EEG-Free Neurological Prognostication After Cardiac Arrest](https://mlsanity.com/paper/2609.18891/) (2026)
- [PPG-LM: A Photoplethysmography-Language Model with Multi-Level Clinical Alignment](https://mlsanity.com/paper/2609.33516/) (2026)
- [HiPPT: Physiology-Grounded Wearable PPG Representation Learning via Multi-Scope Text Alignment](https://mlsanity.com/paper/2610.14489/) (2026)

## Abstract (arXiv)

Electrocardiograms (ECGs) play a vital role in monitoring cardiac health and diagnosing heart diseases. However, traditional deep learning approaches for ECG analysis rely heavily on large-scale manual annotations, which are both time-consuming and resource-intensive to obtain. To overcome this limitation, self-supervised learning (SSL) has emerged as a promising alternative, enabling the extraction of robust ECG representations that can be efficiently transferred to various downstream tasks. While previous studies have explored SSL for ECG pretraining and multi-modal ECG-language alignment, they often fail to capture the multi-scale nature of ECG signals. As a result, these methods struggle to learn generalized representations due to their inability to model the hierarchical structure of ECG data. To address this gap, we introduce MELP, a novel Multi-scale ECG-Language Pretraining (MELP) model that fully leverages hierarchical supervision from ECG-text pairs. MELP first pretrains a cardiology-specific language model to enhance its understanding of clinical text. It then applies three levels of cross-modal supervision-at the token, beat, and rhythm levels-to align ECG signals with textual reports, capturing structured information across different time scales. We evaluate MELP on three public ECG datasets across multiple tasks, including zero-shot ECG classification, linear probing, and transfer learning. Experimental results demonstrate that MELP outperforms existing SSL methods, underscoring its effectiveness and adaptability across diverse clinical applications. Our code is available at https://github.com/HKU-MedAI/MELP.