# Looking Beyond Accuracy: A Holistic Benchmark of ECG Foundation Models

arXiv: https://arxiv.org/abs/2601.21830 · HTML: https://mlsanity.com/paper/2601.21830/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 37 references that are in the map.
- First posted on arXiv 29 Jan 2026.

## Builds on (most cited first, 17 of 17)

- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture](https://mlsanity.com/paper/2301.08243/) (2023)
- [Designing Network Design Spaces](https://mlsanity.com/paper/2003.13678/) (2020)
- [MedGemma Technical Report](https://mlsanity.com/paper/2507.05201/) (2025)
- [ECG-FM: An Open Electrocardiogram Foundation Model](https://mlsanity.com/paper/2408.05178/) (2024)
- [Domain Generalization for Medical Image Analysis: A Review](https://mlsanity.com/paper/2310.08598/) (2023)
- [FairMedFM: Fairness Benchmarking for Medical Imaging Foundation Models](https://mlsanity.com/paper/2407.00983/) (2024)
- [Domain Generalization through Meta-Learning: A Survey](https://mlsanity.com/paper/2404.02785/) (2024)
- [An Electrocardiogram Foundation Model Built on over 10 Million Recordings with External Evaluation across Multiple Domains](https://mlsanity.com/paper/2410.04133/) (2024)
- [Demographic Bias of Expert-Level Vision-Language Foundation Models in Medical Imaging](https://mlsanity.com/paper/2402.14815/) (2024)
- [Self-Supervised Pre-Training with Joint-Embedding Predictive Architecture Boosts ECG Classification Performance](https://mlsanity.com/paper/2410.13867/) (2024)
- [OpenECG: Benchmarking ECG Foundation Models with Public 1.2 Million Records](https://mlsanity.com/paper/2503.00711/) (2025)
- [BenchECG and xECG: a benchmark and baseline for ECG foundation models](https://mlsanity.com/paper/2509.10151/) (2025)
- [Benchmarking ECG FMs: A Reality Check Across Clinical Tasks](https://mlsanity.com/paper/2509.25095/) (2025)
- [Interfacing Foundation Models' Embeddings](https://mlsanity.com/paper/2312.07532/) (2023)
- [An Electrocardiogram Multi-task Benchmark with Comprehensive Evaluations and Insightful Findings](https://mlsanity.com/paper/2512.08954/) (2025)

## Abstract (arXiv)

The electrocardiogram (ECG) is a cost-effective, highly accessible and widely employed diagnostic tool. With the advent of Foundation Models (FMs), the field of AI-assisted ECG interpretation has begun to evolve, as they enable model reuse across different tasks by relying on embeddings. However, to responsibly employ FMs, it is crucial to rigorously assess to which extent the embeddings they produce are generalizable, particularly in error-sensitive domains such as healthcare. Although prior works have already addressed the problem of benchmarking ECG-expert FMs, they focus predominantly on the evaluation of downstream performance. To fill this gap, this study aims to find an in-depth, comprehensive benchmarking framework for FMs, with a specific focus on ECG-expert ones. To this aim, we introduce a benchmark methodology that complements performance-based evaluation with representation-level analysis, leveraging SHAP and UMAP techniques. Furthermore, we rely on the methodology for carrying out an extensive evaluation of several ECG-expert FMs pretrained via state-of-the-art techniques over different cross-continental datasets and data availability settings; this includes ones featuring data scarcity, a fairly common situation in real-world medical scenarios. Experimental results show that our benchmarking protocol provides a rich insight of ECG-expert FMs' embedded patterns, enabling a deeper understanding of their representational structure and generalizability.