# Pretraining Strategies and Scaling for ECG Foundation Models: A Systematic Study

arXiv: https://arxiv.org/abs/2605.12241 · HTML: https://mlsanity.com/paper/2605.12241/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 56 references that are in the map.
- First posted on arXiv 12 May 2026.

## Builds on (most cited first, 7 of 7)

- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Learning General Representation of 12-Lead Electrocardiogram with a Joint-Embedding Predictive Architecture](https://mlsanity.com/paper/2410.08559/) (2024)
- [OpenECG: Benchmarking ECG Foundation Models with Public 1.2 Million Records](https://mlsanity.com/paper/2503.00711/) (2025)
- [BenchECG and xECG: a benchmark and baseline for ECG foundation models](https://mlsanity.com/paper/2509.10151/) (2025)
- [Domain-Adapted Fine-Tuning of ECG Foundation Models for Multi-Label Structural Heart Disease Screening](https://mlsanity.com/paper/2604.23385/) (2026)

## Built on by (most cited first, 1 of 1)

- [Training and Scaling Compute-Optimal Physiological Waveform Foundation Models](https://mlsanity.com/paper/2610.05649/) (2026)

## Abstract (arXiv)

Specialized foundation models are beginning to emerge in various medical subdomains, but pretraining methodologies and parametric scaling with the size of the pretraining dataset are rarely assessed systematically and in a like-for-like manner. This work focuses on foundation models for electrocardiography (ECG) data, one of the most widely captured physiological time series world-wide. We present a comprehensive assessment of pretraining methodologies, covering five different contrastive and non-contrastive self-supervised learning objectives for ECG foundation models, and investigate their scaling behavior with pretraining dataset sizes up to 11M input samples, exclusively from publicly available sources. Pretraining strategy has a meaningful and consistent impact on downstream performance, with contrastive predictive coding (slightly ahead of JEPA) yielding the most transferable representations across diverse clinical tasks. Scaling pretraining data continues to yield meaningful improvements up to 11M samples for most objectives. We also compare model architectures across all pretraining methodologies and find evidence for a clear superiority of structured state space models compared to transformers and CNN models. We hypothesize that the strong inductive biases of structured state space models, rather than pretraining scale alone, are the primary driver of effective ECG representation learning, with important implications for future foundation model development in this and potentially other physiological signal domains.