# CREMA: A Contrastive Regularized Masked Autoencoder for Robust ECG Diagnostics across Clinical Domains

arXiv: https://arxiv.org/abs/2407.07110 · HTML: https://mlsanity.com/paper/2407.07110/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 39 references that are in the map.
- First posted on arXiv 26 Jun 2024.

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [TS2Vec: Towards Universal Representation of Time Series](https://mlsanity.com/paper/2106.10466/) (2021)
- [Unsupervised Scalable Representation Learning for Multivariate Time Series](https://mlsanity.com/paper/1901.10738/) (2019)
- [CLOCS: Contrastive Learning of Cardiac Signals Across Space, Time, and Patients](https://mlsanity.com/paper/2005.13249/) (2020)
- [Contrastive Masked Autoencoders are Stronger Vision Learners](https://mlsanity.com/paper/2207.13532/) (2022)
- [Unsupervised Time-Series Representation Learning with Iterative Bilinear Temporal-Spectral Fusion](https://mlsanity.com/paper/2202.04770/) (2022)
- [MAEEG: Masked Auto-encoder for EEG Representation Learning](https://mlsanity.com/paper/2211.02625/) (2022)
- [Contrast Everything: A Hierarchical Contrastive Framework for Medical Time-Series](https://mlsanity.com/paper/2310.14017/) (2023)
- [TimeMAE: Self-Supervised Representations of Time Series with Decoupled Masked Autoencoders](https://mlsanity.com/paper/2303.00320/) (2023)
- [Self-supervised Learning for ECG-based Emotion Recognition](https://mlsanity.com/paper/1910.07497/) (2019)
- [Neighborhood Contrastive Learning Applied to Online Patient Monitoring](https://mlsanity.com/paper/2106.05142/) (2021)
- [Online false discovery rate control for anomaly detection in time series](https://mlsanity.com/paper/2112.03196/) (2021)
- [Evaluating Self-Supervised Learning via Risk Decomposition](https://mlsanity.com/paper/2302.03068/) (2023)

## Built on by (most cited first, 8 of 8)

- [ECG-FM: An Open Electrocardiogram Foundation Model](https://mlsanity.com/paper/2408.05178/) (2024)
- [From Token to Rhythm: A Multi-Scale Approach for ECG-Language Pretraining](https://mlsanity.com/paper/2506.21803/) (2025)
- [OpenECG: Benchmarking ECG Foundation Models with Public 1.2 Million Records](https://mlsanity.com/paper/2503.00711/) (2025)
- [FoundationalECGNet: A Lightweight Foundational Model for ECG-based Multitask Cardiac Analysis](https://mlsanity.com/paper/2509.08961/) (2025)
- [Position: Evaluation of ECG Representations Must Be Fixed](https://mlsanity.com/paper/2602.17531/) (2026)
- [Extending Pretrained 10-Second ECG Foundation Models to Longer Horizons](https://mlsanity.com/paper/2605.16975/) (2026)
- [TRACE: Expert-Aligned ECG Representation Learning with Rigorous Benchmarking and Real-World Validation in Acute Cardiac Care](https://mlsanity.com/paper/2609.34088/) (2026)
- [PhysioTRACE: Provenance-Aware Stress Tests for Physiological Foundation Models](https://mlsanity.com/paper/2609.34466/) (2026)

## Abstract (arXiv)

Electrocardiogram (ECG) diagnosis remains challenging due to limited labeled data and the need to capture subtle yet clinically meaningful variations in rhythm and morphology. We present CREMA (Contrastive Regularized Masked Autoencoder), a foundation model for 12-lead ECGs designed to learn generalizable representations through self-supervised pretraining. CREMA combines generative learning and contrastive regularization via a Contrastive Regularized MAE loss, and employs a Signal Transformer (SiT) architecture to capture both local waveform details and global temporal dependencies. We evaluate CREMA on benchmark datasets and real-world clinical environments, including deployment scenarios with significant distribution shifts. CREMA outperforms supervised baselines and existing self-supervised models in both linear probing and fine-tuning evaluations. Notably, it maintains superior performance across diverse clinical domains, such as emergency care, highlighting its robustness under real-world conditions. These results demonstrate that CREMA serves as a scalable and reliable foundation model for ECG diagnostics, supporting downstream applications across heterogeneous and high-risk clinical settings.