# ECHOPulse: ECG controlled echocardio-grams video generation

arXiv: https://arxiv.org/abs/2410.03143 · HTML: https://mlsanity.com/paper/2410.03143/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 44 references that are in the map.
- First posted on arXiv 4 Oct 2024.

## Builds on (most cited first, 28 of 28)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Structured Denoising Diffusion Models in Discrete State-Spaces](https://mlsanity.com/paper/2107.03006/) (2021)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [Attention U-Net: Learning Where to Look for the Pancreas](https://mlsanity.com/paper/1804.03999/) (2018)
- [Towards Accurate Generative Models of Video: A New Metric & Challenges](https://mlsanity.com/paper/1812.01717/) (2018)
- [Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval](https://mlsanity.com/paper/2104.00650/) (2021)
- [MaskGIT: Masked Generative Image Transformer](https://mlsanity.com/paper/2202.04200/) (2022)
- [Language Model Beats Diffusion -- Tokenizer is Key to Visual Generation](https://mlsanity.com/paper/2310.05737/) (2023)
- [Phenaki: Variable Length Video Generation From Open Domain Textual Description](https://mlsanity.com/paper/2210.02399/) (2022)
- [VideoComposer: Compositional Video Synthesis with Motion Controllability](https://mlsanity.com/paper/2306.02018/) (2023)
- [MAGVIT: Masked Generative Video Transformer](https://mlsanity.com/paper/2212.05199/) (2022)
- [Deep Learning for Segmentation using an Open Large-Scale Dataset in 2D Echocardiography](https://mlsanity.com/paper/1908.06948/) (2019)
- [Scaling Up Models and Data with textttt5x and textttseqio](https://mlsanity.com/paper/2203.17189/) (2022)
- [Generating Videos with Dynamics-aware Implicit Generative Adversarial Networks](https://mlsanity.com/paper/2202.10571/) (2022)
- [Guiding Masked Representation Learning to Capture Spatio-Temporal Relationship of Electrocardiogram](https://mlsanity.com/paper/2402.09450/) (2024)
- [Improved Masked Image Generation with Token-Critic](https://mlsanity.com/paper/2209.04439/) (2022)
- [Moonshot: Towards Controllable Video Generation and Editing with Multimodal Conditions](https://mlsanity.com/paper/2401.01827/) (2024)
- [On Mean Absolute Error for Deep Neural Network Based Vector-to-Vector Regression](https://mlsanity.com/paper/2008.07281/) (2020)
- [Feature-Conditioned Cascaded Video Diffusion Models for Precise Echocardiogram Synthesis](https://mlsanity.com/paper/2303.12644/) (2023)
- [EchoNet-Synthetic: Privacy-preserving Video Generation for Safe Medical Data Sharing](https://mlsanity.com/paper/2406.00808/) (2024)
- [Vision-Language Synthetic Data Enhances Echocardiography Downstream Tasks](https://mlsanity.com/paper/2403.19880/) (2024)
- [Explainable and Controllable Motion Curve Guided Cardiac Ultrasound Video Generation](https://mlsanity.com/paper/2407.21490/) (2024)

## Built on by (most cited first, 9 of 9)

- [EchoFM: Foundation Model for Generalizable Echocardiogram Analysis](https://mlsanity.com/paper/2410.23413/) (2024)
- [Towards Next-Generation Medical Agent: How o1 is Reshaping Decision-Making in Medical Scenarios](https://mlsanity.com/paper/2411.14461/) (2024)
- [Foundation Models for Low-Resource Language Education (Vision Paper)](https://mlsanity.com/paper/2412.04774/) (2024)
- [Legal Evalutions and Challenges of Large Language Models](https://mlsanity.com/paper/2411.10137/) (2024)
- [RAU: Reference-based Anatomical Understanding with Vision Language Models](https://mlsanity.com/paper/2509.22404/) (2025)
- [RSTR: Reducing SpatioTemporal Redundancy in Diffusion Transformers](https://mlsanity.com/paper/2512.14096/) (2025)
- [Echo-Path: Pathology-Conditioned Echo Video Generation](https://mlsanity.com/paper/2509.17190/) (2025)
- [Chain of Flow: ECG-Conditioned 4D Cardiac Cine Generation from Patient-Specific Anatomical Anchor](https://mlsanity.com/paper/2602.22919/) (2026)
- [Focus, Align, and Diffuse: Time-Series-Aware Keyframe Diffusion for Cardiac Dynamic Synthesis from Sparse Observations](https://mlsanity.com/paper/2610.19050/) (2026)

## Abstract (arXiv)

Echocardiography (ECHO) is essential for cardiac assessments, but its video quality and interpretation heavily relies on manual expertise, leading to inconsistent results from clinical and portable devices. ECHO video generation offers a solution by improving automated monitoring through synthetic data and generating high-quality videos from routine health data. However, existing models often face high computational costs, slow inference, and rely on complex conditional prompts that require experts' annotations. To address these challenges, we propose ECHOPULSE, an ECG-conditioned ECHO video generation model. ECHOPULSE introduces two key advancements: (1) it accelerates ECHO video generation by leveraging VQ-VAE tokenization and masked visual token modeling for fast decoding, and (2) it conditions on readily accessible ECG signals, which are highly coherent with ECHO videos, bypassing complex conditional prompts. To the best of our knowledge, this is the first work to use time-series prompts like ECG signals for ECHO video generation. ECHOPULSE not only enables controllable synthetic ECHO data generation but also provides updated cardiac function information for disease monitoring and prediction beyond ECG alone. Evaluations on three public and private datasets demonstrate state-of-the-art performance in ECHO video generation across both qualitative and quantitative measures. Additionally, ECHOPULSE can be easily generalized to other modality generation tasks, such as cardiac MRI, fMRI, and 3D CT generation. Demo can seen from urlhttps://github.com/levyisthebest/ECHOPulsePrelease.