mlsanityData to 14 Oct 2026

GEM: Empowering MLLM for Grounded ECG Understanding with Time Series and Images

arXiv 2503.06073 ↗First posted 8 Mar 2025cs.CL cs.AI cs.CVcited by 39

Not on the map. Why?

Abstract (arXiv)

While recent multimodal large language models (MLLMs) have advanced automated ECG interpretation, they still face two key limitations: (1) insufficient multimodal synergy between time series signals and visual ECG representations, and (2) limited explainability in linking diagnoses to granular waveform evidence. We introduce GEM, the first MLLM unifying ECG time series, 12-lead ECG images and text for grounded and clinician-aligned ECG interpretation. GEM enables feature-grounded analysis, evidence-driven reasoning, and a clinician-like diagnostic process through three core innovations: a dual-encoder framework extracting complementary time series and image features, cross-modal alignment for effective multimodal understanding, and knowledge-guided instruction generation for generating high-granularity grounding data (ECG-Grounding) linking diagnoses to measurable parameters (e.g., QRS/PR Intervals). Additionally, we propose the Grounded ECG Understanding task, a clinically motivated benchmark designed to comprehensively assess the MLLM's capability in grounded ECG understanding. Experimental results on both existing and our proposed benchmarks show GEM significantly improves predictive performance (CSN 7.4% uparrow), explainability (22.7% uparrow), and grounding (24.8% uparrow), making it more suitable for real-world clinical applications. GitHub repository: https://github.com/lanxiang1017/GEM.git

Key facts. Cited by 39 papers in the map (data through 14 Oct 2026). Builds on 27 of its 68 references that are in the map. First posted on arXiv 8 Mar 2025.

Builds on

27 of 68 references.

References · 27

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023Visual Instruction Tuningcited by 7,869
  5. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  7. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  8. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  9. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,305
  10. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 906
  11. 2023VideoChat: Chat-Centric Video Understandingcited by 857
  12. 2023Vision-Language Models for Vision Tasks: A Surveycited by 839
15 more
  1. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 608
  2. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  3. 2023MultiModal-GPT: A Vision and Language Model for Dialogue with Humanscited by 255
  4. 2023Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integrationcited by 172
  5. 2020CLOCS: Contrastive Learning of Cardiac Signals Across Space, Time, and Patientscited by 171
  6. 2023A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethicscited by 163
  7. 2023DetGPT: Detect What You Need via Reasoningcited by 112
  8. 2024ECG-FM: An Open Electrocardiogram Foundation Modelcited by 90
  9. 2024Health-LLM: Large Language Models for Health Prediction via Wearable Sensor Datacited by 66
  10. 2023Frozen Language Model Helps ECG Zero-Shot Learningcited by 62
  11. 2024ECG-Chat: A Large ECG-Language Model for Cardiac Disease Diagnosiscited by 38
  12. 2021Intra-Inter Subject Self-supervised Learning for Multivariate Cardiac Signalscited by 35
  13. 2024Teach Multimodal LLMs to Comprehend Electrocardiographic Imagescited by 28
  14. 2023ECG-Image-Kit: A Synthetic Image Generation Toolbox to Facilitate Deep Learning-Based Electrocardiogram Digitizationcited by 20
  15. 2023Towards Enhancing Time Series Contrastive Learning: A Dynamic Bad Pair Mining Approachcited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 11; 2026: 28.

39 papers.

Sort:

By area:

  1. 2025R1-VL: Learning to Reason with Multimodal Large Language Models via Step-wise Group Relative Policy Optimizationcited by 301
  2. 2025R1-ShareVL: Incentivizing Reasoning Capability of Multimodal Large Language Models via Share-GRPOcited by 52
  3. 2025QoQ-Med: Building Multimodal Clinical Foundation Models with Domain-Aware GRPO Trainingcited by 38
  4. 2025TimeOmni-1: Incentivizing Complex Reasoning with Time Series in Large Language Modelscited by 27
  5. 2025A Survey on Agentic Multimodal Large Language Modelscited by 23
  6. 2025SATORI-R1: Incentivizing Multimodal Reasoning through Explicit Visual Anchoringcited by 17
  7. 2026ECG-R1: Protocol-Guided and Modality-Agnostic MLLM for Reliable ECG Interpretationcited by 9
  8. 2025MedTVT-R1: A Multimodal LLM Empowering Medical Reasoning and Diagnosiscited by 7
  9. 2025Multimodal LLM-Guided Semantic Correction in Text-to-Image Diffusioncited by 5
  10. 2025UniECG: Understanding and Generating ECG in One Unified Modelcited by 4
  11. 2026ECG-Agent: On-Device Tool-Calling Agent for ECG Multi-Turn Dialoguecited by 4
  12. 2026ECG-Reasoning-Benchmark: A Benchmark for Evaluating Clinical Reasoning Capabilities in ECG Interpretationcited by 4
27 more
  1. 2025MEETI: A Multimodal ECG Dataset from MIMIC-IV-ECG with Signals, Images, Features and Interpretationscited by 3
  2. 2026Time Series Reasoning via Process-Verifiable Thinking Data Synthesis and Scheduling for Tailored LLM Reasoningcited by 3
  3. 2026How Well Do Multimodal Models Reason on ECG Signals?cited by 3
  4. 2026TiWeaver: Unified Temporal Dynamics Modeling via Contextual Patchingcited by 2
  5. 2025SpiroLLM: Finetuning Pretrained LLMs to Understand Spirogram Time Series with Clinical Validation in COPD Reportingcited by 1
  6. 2026TS-MLLM: A Multi-Modal Large Language Model-based Framework for Industrial Time-Series Big Data Analysiscited by 1
  7. 2026DeepArrhythmia: Segment-Contextualized ECG Arrhythmia Classification via Selective Evidence Acquisitioncited by 1
  8. 2026LVCG: Learning ECG Representations in the Latent Vectorcardiogram Spacecited by 1
  9. 2026ECG Mirage: Revealing and Mitigating the Underutilisation of ECGs in Vision-Language Models for Clinical Predictioncited by 1
  10. 2026Towards Efficient and Robust Linguistic Emotion Diagnosis for Mental Health via Multi-Agent Instruction Refinementcited by 0
  11. 2026R1-SyntheticVL: Is Synthetic Data from Generative Models Ready for Multimodal Large Language Model?cited by 0
  12. 2026NeuroNarrator: A Generalist EEG-to-Text Foundation Model for Clinical Interpretation via Spectro-Spatial Grounding and Temporal State-Space Reasoningcited by 0
  13. 2026Reasoning Before Diagnosis: Physician-Inspired Structured Thinking for ECG Classificationcited by 0
  14. 2026EVL-ECG: Efficient ECG Interpretation With Multi-Aspect Heterogeneous Knowledge Distillationcited by 0
  15. 2026ATRIA: Adaptive Traceable ECG Reporting with Iterative Agentscited by 0
  16. 2026VLT: A Vision-Language-Time Series Multimodal Foundation Model for Industrial Intelligencecited by 0
  17. 2026Cardiologent: Multi-Agent Clinical Decision Support for Patient-Level Arrhythmia Assessment, Urgency, and Managementcited by 0
  18. 2026Diagnosing as Cardiologists Do: ECG Agents with Doctor-Grounded Priors for Clinical Reasoning Across Diseases and Populationscited by 0
  19. 2026NeuroECG: ECGFounder-Based Deep ECG Representation for EEG-Free Neurological Prognostication After Cardiac Arrestcited by 0
  20. 2026PhysioBench: A Unified Benchmark for Physiological Signal Question Answeringcited by 0
  21. 2026Large language models in medical time series analysiscited by 0
  22. 2026ECG-Scroll: A Long-Horizon, Streaming Benchmark and Agent Environment for Interpretation of Ambulatory Electrocardiogramscited by 0
  23. 2026Robust Transfer Learning for Paper ECG Recognitioncited by 0
  24. 2026Retrieval-Augmented Multimodal Language Model with Heterogeneous Evidence Integration for Clinical Prognosiscited by 0
  25. 2026HiPPT: Physiology-Grounded Wearable PPG Representation Learning via Multi-Scope Text Alignmentcited by 0
  26. 2026ECG-RSI: Recursive Self-Improvement through Targeted Electrocardiogram Waveform Evolutioncited by 0
  27. 2026Lotus: Learning to Understand Time-series by Learning to Forecastcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Enhancing Explainable Cardiac Diagnosis with Guide-Grounded Multimodal LLMs
  2. 2026ECGQuest: Benchmarking and Fine-Tuning Language Models for Electrocardiography
  3. 2024De-biased Multimodal Electrocardiogram Analysis
  4. 2025anyECG-chat: A Generalist ECG-MLLM for Flexible ECG Input and Multi-Task Understanding
  5. 2026Holtercare-Bench: A Multimodal Benchmark for Evaluating Long-Term Dynamic ECG Analysis
  6. 2025ECG-aBcDe: Overcoming Model Dependence, Encoding ECG into a Universal Language for Any LLM
  7. 2025Interpretable Multimodal Zero-Shot ECG Diagnosis via Structured Clinical Knowledge Alignment
  8. 2025HeartcareGPT: A Unified Multimodal ECG Suite for Dual Signal-Image Modeling and Understanding