# GEMTrans: A General, Echocardiography-based, Multi-Level Transformer Framework for Cardiovascular Diagnosis

arXiv: https://arxiv.org/abs/2308.13217 · HTML: https://mlsanity.com/paper/2308.13217/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 26 references that are in the map.
- First posted on arXiv 25 Aug 2023.

## Builds on (most cited first, 11 of 11)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [ProtoPFormer: Concentrating on Prototypical Parts in Vision Transformers for Interpretable Image Recognition](https://mlsanity.com/paper/2208.10431/) (2022)
- [High-Throughput Precision Phenotyping of Left Ventricular Hypertrophy with Cardiovascular Deep Learning](https://mlsanity.com/paper/2106.12511/) (2021)
- [Ultrasound Video Transformers for Cardiac Ejection Fraction Estimation](https://mlsanity.com/paper/2107.00977/) (2021)
- [Supervising Model Attention with Human Explanations for Robust Natural Language Inference](https://mlsanity.com/paper/2104.08142/) (2021)
- [EchoGNN: Explainable Ejection Fraction Estimation with Graph Neural Networks](https://mlsanity.com/paper/2208.14003/) (2022)
- [A New Semi-supervised Learning Benchmark for Classifying View and Diagnosing Aortic Stenosis from Echocardiograms](https://mlsanity.com/paper/2108.00080/) (2021)
- [Light-weight spatio-temporal graphs for segmentation and ejection fraction prediction in cardiac ultrasound](https://mlsanity.com/paper/2207.02549/) (2022)

## Built on by (most cited first, 9 of 9)

- [EchoNarrator: Generating natural text explanations for ejection fraction predictions](https://mlsanity.com/paper/2410.23744/) (2024)
- [ProtoEFNet: Dynamic Prototype Learning for Inherently Interpretable Ejection Fraction Estimation in Echocardiography](https://mlsanity.com/paper/2512.03339/) (2025)
- [Anatomically Constrained Transformers for Cardiac Amyloidosis Classification](https://mlsanity.com/paper/2509.19691/) (2025)
- [Beyond Independent Frames: Latent Attention Masked Autoencoders for Multi-View Echocardiography](https://mlsanity.com/paper/2604.15096/) (2026)
- [Echo-E3Net: Efficient Endocardial Spatio-Temporal Network for Ejection Fraction Estimation](https://mlsanity.com/paper/2503.17543/) (2025)
- [Mamba-3D as Masked Autoencoders for Accurate and Data-Efficient Analysis of Medical Ultrasound Videos](https://mlsanity.com/paper/2503.20258/) (2025)
- [Anatomically Constrained Transformers for Echocardiogram Analysis](https://mlsanity.com/paper/2511.01109/) (2025)
- [Optimizing Point-of-Care Ultrasound Video Acquisition for Probabilistic Multi-Task Heart Failure Detection](https://mlsanity.com/paper/2602.13658/) (2026)
- [Reading the Whole Heart: Latent-Attention Masked Autoencoders for Multimodal Cardiac Representation Learning](https://mlsanity.com/paper/2609.12035/) (2026)

## Abstract (arXiv)

Echocardiography (echo) is an ultrasound imaging modality that is widely used for various cardiovascular diagnosis tasks. Due to inter-observer variability in echo-based diagnosis, which arises from the variability in echo image acquisition and the interpretation of echo images based on clinical experience, vision-based machine learning (ML) methods have gained popularity to act as secondary layers of verification. For such safety-critical applications, it is essential for any proposed ML method to present a level of explainability along with good accuracy. In addition, such methods must be able to process several echo videos obtained from various heart views and the interactions among them to properly produce predictions for a variety of cardiovascular measurements or interpretation tasks. Prior work lacks explainability or is limited in scope by focusing on a single cardiovascular task. To remedy this, we propose a General, Echo-based, Multi-Level Transformer (GEMTrans) framework that provides explainability, while simultaneously enabling multi-video training where the inter-play among echo image patches in the same frame, all frames in the same video, and inter-video relationships are captured based on a downstream task. We show the flexibility of our framework by considering two critical tasks including ejection fraction (EF) and aortic stenosis (AS) severity detection. Our model achieves mean absolute errors of 4.15 and 4.84 for single and dual-video EF estimation and an accuracy of 96.5 % for AS detection, while providing informative task-specific attention maps and prototypical explainability.