# Benchmarking ECG FMs: A Reality Check Across Clinical Tasks

arXiv: https://arxiv.org/abs/2509.25095 · HTML: https://mlsanity.com/paper/2509.25095/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 58 references that are in the map.
- First posted on arXiv 29 Sep 2025.

## Builds on (most cited first, 30 of 31)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality](https://mlsanity.com/paper/2405.21060/) (2024)
- [A Survey on Evaluation of Large Language Models](https://mlsanity.com/paper/2307.03109/) (2023)
- [Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture](https://mlsanity.com/paper/2301.08243/) (2023)
- [Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depth](https://mlsanity.com/paper/2010.15327/) (2020)
- [Automatic diagnosis of the 12-lead ECG using a deep neural network](https://mlsanity.com/paper/1904.01949/) (2019)
- [Deep Learning for ECG Analysis: Benchmarks and Insights from PTB-XL](https://mlsanity.com/paper/2004.13701/) (2020)
- [Towards Foundation Models for Scientific Machine Learning: Characterizing Scaling and Transfer Behavior](https://mlsanity.com/paper/2306.00258/) (2023)
- [ECG-FM: An Open Electrocardiogram Foundation Model](https://mlsanity.com/paper/2408.05178/) (2024)
- [Guiding Masked Representation Learning to Capture Spatio-Temporal Relationship of Electrocardiogram](https://mlsanity.com/paper/2402.09450/) (2024)
- [Battle of the Backbones: A Large-Scale Comparison of Pretrained Models across Computer Vision Tasks](https://mlsanity.com/paper/2310.19909/) (2023)
- [Zero-Shot ECG Classification with Multimodal Learning and Test-time Clinical Knowledge Enhancement](https://mlsanity.com/paper/2403.06659/) (2024)
- [Mamba-360: Survey of State Space Models as Transformer Alternative for Long Sequence Modelling: Methods, Applications, and Challenges](https://mlsanity.com/paper/2404.16112/) (2024)
- [A Closer Look at AUROC and AUPRC under Class Imbalance](https://mlsanity.com/paper/2401.06091/) (2024)
- [Benchmarking foundation models as feature extractors for weakly-supervised computational pathology](https://mlsanity.com/paper/2408.15823/) (2024)
- [EEG-FM-Bench: A Comprehensive Benchmark for the Systematic Evaluation and Diagnostic Analyses of EEG Foundation Models](https://mlsanity.com/paper/2508.17742/) (2025)
- [Learning General Representation of 12-Lead Electrocardiogram with a Joint-Embedding Predictive Architecture](https://mlsanity.com/paper/2410.08559/) (2024)
- [Benchmarking Pathology Foundation Models: Adaptation Strategies and Scenarios](https://mlsanity.com/paper/2410.16038/) (2024)
- [Towards quantitative precision for ECG analysis: Leveraging state space models, self-supervision and patient metadata](https://mlsanity.com/paper/2308.15291/) (2023)
- [Enhancing clinical decision support with physiological waveforms -- a multimodal benchmark in emergency care](https://mlsanity.com/paper/2407.17856/) (2024)
- [Beyond Scalars: Concept-Based Alignment Analysis in Vision Transformers](https://mlsanity.com/paper/2412.06639/) (2024)
- [Advancing Intelligent Sequence Modeling: Evolution, Trade-offs, and Applications of State-Space Architectures from S4 to Mamba](https://mlsanity.com/paper/2503.18970/) (2025)
- [S4Sleep: Elucidating the design space of deep-learning-based sleep stage classification models](https://mlsanity.com/paper/2310.06715/) (2023)
- [Abnormality Prediction and Forecasting of Laboratory Values from Electrocardiogram Signals Using Multimodal Deep Learning](https://mlsanity.com/paper/2411.14886/) (2024)

## Built on by (most cited first, 7 of 7)

- [A Multimodal Deep Learning Framework for Predicting ICU Deterioration: Integrating ECG Waveforms with Clinical Data and Clinician Benchmarking](https://mlsanity.com/paper/2601.06645/) (2026)
- [Looking Beyond Accuracy: A Holistic Benchmark of ECG Foundation Models](https://mlsanity.com/paper/2601.21830/) (2026)
- [Do ECG Foundation Models Transfer to Rare Cardiac Diseases? Evidence from Brugada Syndrome Detection](https://mlsanity.com/paper/2607.03009/) (2026)
- [Explainable Diabetic Retinopathy Classification Using Vision Foundation Models](https://mlsanity.com/paper/2608.28207/) (2026)
- [On the role of the tokenizer in ECG transformer models](https://mlsanity.com/paper/2609.15433/) (2026)
- [When Does Unsupervised Learning Succeed or Fail? A PoS Perspective on Reconstruction-Based Anomaly Detection](https://mlsanity.com/paper/2609.28832/) (2026)
- [PPG-Bench: A Unified Benchmark for PPG Foundation Models](https://mlsanity.com/paper/2610.13435/) (2026)

## Abstract (arXiv)

The 12-lead electrocardiogram (ECG) is a long-standing diagnostic tool. Yet machine learning for ECG interpretation remains fragmented, often limited to narrow tasks or datasets. FMs promise broader adaptability, but fundamental questions remain: Which architectures generalize best? How do models scale with limited labels? What explains performance differences across model families? We benchmarked eight ECG FMs on 26 clinically relevant tasks using 12 public datasets comprising 1,650 regression and classification targets. Models were evaluated under fine-tuning and frozen settings, with scaling analyses across dataset sizes. Results show heterogeneous performance across domains: in adult ECG interpretation, three FMs consistently outperformed strong supervised baselines. In contrast, ECG-CPC, a compact structured state-space model, dominated 5 of 7 task categories, demonstrating that architecture matters more than scale. FMs improved label efficiency 3.3-9x over supervised baselines, though scaling behaviors varied across architectures. Representation analysis reveals that models with similar performance learn markedly different internal structures, suggesting multiple viable paths to effective ECG representation. Overall, while FMs show promise for adult ECG analysis, substantial gaps remain in cardiac structure, outcome prediction, and patient characterization. ECG-CPC's strong performance despite being orders of magnitude smaller challenges the assumption that FM quality requires massive scale, highlighting architectural inductive biases as an untapped opportunity.