# ECG-Soup: Harnessing Multi-Layer Synergy for ECG Foundation Models

arXiv: https://arxiv.org/abs/2509.00102 · HTML: https://mlsanity.com/paper/2509.00102/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 59 references that are in the map.
- First posted on arXiv 27 Aug 2025.

## Builds on (most cited first, 30 of 35)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Do Vision Transformers See Like Convolutional Neural Networks?](https://mlsanity.com/paper/2108.08810/) (2021)
- [Layer by Layer: Uncovering Hidden Representations in Language Models](https://mlsanity.com/paper/2502.02013/) (2025)
- [Representation Degeneration Problem in Training Natural Language Generation Models](https://mlsanity.com/paper/1907.12009/) (2019)
- [A Survey on Mixture of Experts in Large Language Models](https://mlsanity.com/paper/2407.06204/) (2024)
- [CLOCS: Contrastive Learning of Cardiac Signals Across Space, Time, and Patients](https://mlsanity.com/paper/2005.13249/) (2020)
- [Automatic diagnosis of the 12-lead ECG using a deep neural network](https://mlsanity.com/paper/1904.01949/) (2019)
- [Deep Learning for ECG Analysis: Benchmarks and Insights from PTB-XL](https://mlsanity.com/paper/2004.13701/) (2020)
- [Anti-Oversmoothing in Deep Vision Transformers via the Fourier Domain Analysis: From Theory to Practice](https://mlsanity.com/paper/2203.05962/) (2022)
- [Self-supervised ECG Representation Learning for Emotion Recognition](https://mlsanity.com/paper/2002.03898/) (2020)
- [Guiding Masked Representation Learning to Capture Spatio-Temporal Relationship of Electrocardiogram](https://mlsanity.com/paper/2402.09450/) (2024)
- [Self-supervised representation learning from 12-lead ECG data](https://mlsanity.com/paper/2103.12676/) (2021)
- [Whisper-AT: Noise-Robust Automatic Speech Recognizers are Also Strong General Audio Event Taggers](https://mlsanity.com/paper/2307.03183/) (2023)
- [Zero-Shot ECG Classification with Multimodal Learning and Test-time Clinical Knowledge Enhancement](https://mlsanity.com/paper/2403.06659/) (2024)
- [Frozen Language Model Helps ECG Zero-Shot Learning](https://mlsanity.com/paper/2303.12311/) (2023)
- [Visual Query Tuning: Towards Effective Usage of Intermediate Representations for Parameter and Memory Efficient Transfer Learning](https://mlsanity.com/paper/2212.03220/) (2022)
- [Towards Understanding Mixture of Experts in Deep Learning](https://mlsanity.com/paper/2208.02813/) (2022)
- [Self-Supervised Time Series Representation Learning via Cross Reconstruction Transformer](https://mlsanity.com/paper/2205.09928/) (2022)
- [Vision Transformers with Patch Diversification](https://mlsanity.com/paper/2104.12753/) (2021)
- [3KG: Contrastive Learning of 12-Lead Electrocardiograms using Physiologically-Inspired Augmentations](https://mlsanity.com/paper/2106.04452/) (2021)
- [Mitigating Over-smoothing in Transformers via Regularized Nonlocal Functionals](https://mlsanity.com/paper/2312.00751/) (2023)
- [Analyzing Vision Transformers for Image Classification in Class Embedding Space](https://mlsanity.com/paper/2310.18969/) (2023)
- [Enriched CNN-Transformer Feature Aggregation Networks for Super-Resolution](https://mlsanity.com/paper/2203.07682/) (2022)
- [Learning General Representation of 12-Lead Electrocardiogram with a Joint-Embedding Predictive Architecture](https://mlsanity.com/paper/2410.08559/) (2024)
- [Self-Supervised Pre-Training with Joint-Embedding Predictive Architecture Boosts ECG Classification Performance](https://mlsanity.com/paper/2410.13867/) (2024)
- [Real-Time Patient-Specific ECG Classification by 1D Self-Operational Neural Networks](https://mlsanity.com/paper/2110.02215/) (2021)
- [Towards Enhancing Time Series Contrastive Learning: A Dynamic Bad Pair Mining Approach](https://mlsanity.com/paper/2302.03357/) (2023)

## Built on by (most cited first, 1 of 1)

- [Information-theoretic Multimodal Representation Learning for Electrocardiogram Signals](https://mlsanity.com/paper/2605.27583/) (2026)

## Abstract (arXiv)

Transformer-based foundation models for Electrocardiograms (ECGs) have recently achieved impressive performance in many downstream applications.