# ECG-LLM -- training and evaluation of domain-specific large language models for electrocardiography

arXiv: https://arxiv.org/abs/2510.18339 · HTML: https://mlsanity.com/paper/2510.18339/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Medical and biomedical RAG, one of 15 directions in Retrieval-augmented generation.
- Builds on 30 of its 57 references that are in the map.
- First posted on arXiv 21 Oct 2025.

## Placed in

- Retrieval-augmented generation / [Medical and biomedical RAG](https://mlsanity.com/rag/medical-and-biomedical-rag/)

## Builds on (most cited first, 30 of 30)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions](https://mlsanity.com/paper/2311.05232/) (2023)
- [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://mlsanity.com/paper/2403.04132/) (2024)
- [Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing](https://mlsanity.com/paper/2007.15779/) (2020)
- [A Comprehensive Overview of Large Language Models](https://mlsanity.com/paper/2307.06435/) (2023)
- [From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline](https://mlsanity.com/paper/2406.11939/) (2024)
- [Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models](https://mlsanity.com/paper/2405.01535/) (2024)
- [AlignScore: Evaluating Factual Consistency with a Unified Alignment Function](https://mlsanity.com/paper/2305.16739/) (2023)
- [MinerU: An Open-Source Solution for Precise Document Content Extraction](https://mlsanity.com/paper/2409.18839/) (2024)
- [OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Data](https://mlsanity.com/paper/2410.01560/) (2024)
- [InvestLM: A Large Language Model for Investment using Financial Domain Instruction Tuning](https://mlsanity.com/paper/2309.13064/) (2023)
- [ECG-Chat: A Large ECG-Language Model for Cardiac Disease Diagnosis](https://mlsanity.com/paper/2408.08849/) (2024)
- [How Much Can RAG Help the Reasoning of LLM?](https://mlsanity.com/paper/2410.02338/) (2024)
- [Medical Adaptation of Large Language and Vision-Language Models: Are We Making Progress?](https://mlsanity.com/paper/2411.04118/) (2024)
- [SFT Doesn't Always Hurt General Capabilities: Revisiting Domain-Specific Fine-Tuning in LLMs](https://mlsanity.com/paper/2509.20758/) (2025)
- [Can One Domain Help Others? A Data-Centric Study on Multi-Domain Reasoning via Reinforcement Learning](https://mlsanity.com/paper/2507.17512/) (2025)
- [From Text to Insight: Large Language Models for Materials Science Data Extraction](https://mlsanity.com/paper/2407.16867/) (2024)
- [Enhancing Clinical Efficiency through LLM: Discharge Note Generation for Cardiac Patients](https://mlsanity.com/paper/2404.05144/) (2024)
- [Q-Heart: ECG Question Answering via Knowledge-Informed Multimodal LLMs](https://mlsanity.com/paper/2505.06296/) (2025)
- [FineTuneBench: How well do commercial fine-tuning APIs infuse knowledge into LLMs?](https://mlsanity.com/paper/2411.05059/) (2024)
- [Zodiac: A Cardiologist-Level LLM Framework for Multi-Agent Diagnostics](https://mlsanity.com/paper/2410.02026/) (2024)
- [From Style to Facts: Mapping the Boundaries of Knowledge Injection with Finetuning](https://mlsanity.com/paper/2503.05919/) (2025)
- [MIRIAD: Augmenting LLMs with millions of medical query-response pairs](https://mlsanity.com/paper/2506.06091/) (2025)
- [RadPhi-3: Small Language Models for Radiology](https://mlsanity.com/paper/2411.13604/) (2024)
- [Question answering systems for health professionals at the point of care -- a systematic review](https://mlsanity.com/paper/2402.01700/) (2024)
- [Fine-Tuning LLMs on Small Medical Datasets: Text Classification and Normalization Effectiveness on Cardiology reports and Discharge records](https://mlsanity.com/paper/2503.21349/) (2025)
- [CardAIc-Agents: A Multimodal Framework with Hierarchical Adaptation for Cardiac Care Support](https://mlsanity.com/paper/2508.13256/) (2025)

## Abstract (arXiv)

Domain-adapted open-weight large language models (LLMs) offer promising healthcare applications, from queryable knowledge bases to multimodal assistants, with the crucial advantage of local deployment for privacy preservation. However, optimal adaptation strategies, evaluation methodologies, and performance relative to general-purpose LLMs remain poorly characterized. We investigated these questions in electrocardiography, an important area of cardiovascular medicine, by finetuning open-weight models on domain-specific literature and implementing a multi-layered evaluation framework comparing finetuned models, retrieval-augmented generation (RAG), and Claude Sonnet 3.7 as a representative general-purpose model. Finetuned Llama 3.1 70B achieved superior performance on multiple-choice evaluations and automatic text metrics, ranking second to Claude 3.7 in LLM-as-a-judge assessments. Human expert evaluation favored Claude 3.7 and RAG approaches for complex queries. Finetuned models significantly outperformed their base counterparts across nearly all evaluation modes. Our findings reveal substantial performance heterogeneity across evaluation methodologies, underscoring assessment complexity. Nevertheless, domain-specific adaptation through finetuning and RAG achieves competitive performance with proprietary models, supporting the viability of privacy-preserving, locally deployable clinical solutions.