mlsanityData to 14 Oct 2026

MedAlign: A Clinician-Generated Dataset for Instruction Following with Electronic Medical Records

arXiv 2308.14089 ↗First posted 27 Aug 2023cs.CL cs.AI cs.LGcited by 56

Not on the map. Why?

Abstract (arXiv)

The ability of large language models (LLMs) to follow natural language instructions with human-level fluency suggests many opportunities in healthcare to reduce administrative burden and improve quality of care. However, evaluating LLMs on realistic text generation tasks for healthcare remains challenging. Existing question answering datasets for electronic health record (EHR) data fail to capture the complexity of information needs and documentation burdens experienced by clinicians. To address these challenges, we introduce MedAlign, a benchmark dataset of 983 natural language instructions for EHR data. MedAlign is curated by 15 clinicians (7 specialities), includes clinician-written reference responses for 303 instructions, and provides 276 longitudinal EHRs for grounding instruction-response pairs. We used MedAlign to evaluate 6 general domain LLMs, having clinicians rank the accuracy and quality of each LLM response. We found high error rates, ranging from 35% (GPT-4) to 68% (MPT-7B-Instruct), and an 8.3% drop in accuracy moving from 32k to 2k context lengths for GPT-4. Finally, we report correlations between clinician rankings and automated natural language generation metrics as a way to rank LLMs without human review. We make MedAlign available under a research data use agreement to enable LLM evaluations on tasks aligned with clinician needs and preferences.

Key facts. Cited by 56 papers in the map (data through 14 Oct 2026). Builds on 30 of its 70 references that are in the map. First posted on arXiv 27 Aug 2023.

Builds on

30 of 70 references.

References · 30

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  4. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  5. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  6. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  7. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  8. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  9. 2023A Survey of Large Language Modelscited by 2,576
  10. 2022Large Language Models Encode Clinical Knowledgecited by 1,830
  11. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
  12. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,356
18 more
  1. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,288
  2. 2020COMET: A Neural Framework for MT Evaluationcited by 1,033
  3. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 740
  4. 2023Capabilities of GPT-4 on Medical Challenge Problemscited by 638
  5. 2023Textbooks Are All You Needcited by 569
  6. 2022MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answeringcited by 543
  7. 2023MedAlpaca -- An Open-Source Collection of Medical Conversational AI Models and Training Datacited by 286
  8. 2022Towards a Unified Multi-Dimensional Evaluator for Text Generationcited by 264
  9. 2022LinkBERT: Pretraining Language Models with Document Linkscited by 211
  10. 2018emrQA: A Large Corpus for Question Answering on Electronic Medical Recordscited by 135
  11. 2023Clinical Camel: An Open Expert-Level Medical Language Model with Dialogue-Based Knowledge Encodingcited by 71
  12. 2021Compression, Transduction, and Creation: A Unified Framework for Evaluating Natural Language Generationcited by 60
  13. 2023DERA: Enhancing Large Language Model Completions with Dialog-Enabled Resolving Agentscited by 52
  14. 2020CliniQG4QA: Generating Diverse Questions for Domain Adaptation of Clinical Question Answeringcited by 26
  15. 2020Clinical Reading Comprehension: A Thorough Analysis of the emrQA Datasetcited by 23
  16. 2022Learning to Ask Like a Physiciancited by 16
  17. 2023NLG Evaluation Metrics Beyond Correlation Analysis: An Empirical Metric Preference Checklistcited by 16
  18. 2018Annotating Electronic Medical Records for Question Answeringcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 8; 2024: 18; 2025: 15; 2026: 15.

56 papers.

Sort:

By area:

  1. 2025HealthBench: Evaluating Large Language Models Towards Improved Human Healthcited by 429
  2. 2023Adapted Large Language Models Can Outperform Medical Experts in Clinical Text Summarizationcited by 246
  3. 2024Capabilities of Gemini Models in Medicinecited by 237
  4. 2023A Survey of Large Language Models in Medicine: Progress, Application, and Challengecited by 124
  5. 2024Advancing Multimodal Medical Capabilities of Geminicited by 90
  6. 2024Towards a clinically accessible radiology foundation model: open-access and lightweight, with automated evaluationcited by 51
  7. 2024Towards Evaluating and Building Versatile Large Language Models for Medicinecited by 40
  8. 2025MedCaseReasoning: Evaluating and learning diagnostic reasoning from clinical case reportscited by 36
  9. 2024Zero-Shot Clinical Trial Patient Matching with LLMscited by 35
  10. 2024EHRNoteQA: An LLM Benchmark for Real-World Clinical Practice Using Discharge Summariescited by 35
  11. 2025BIOMEDICA: An Open Biomedical Image-Caption Archive, Dataset, and Vision-Language Models Derived from Scientific Literaturecited by 34
  12. 2025TIMER: Temporal Instruction Modeling and Evaluation for Longitudinal Clinical Recordscited by 34
38 more
  1. 2023Clinfo.ai: An Open-Source Retrieval-Augmented Large Language Model System for Answering Medical Questions using Scientific Literaturecited by 29
  2. 2024Foundation Models in Radiology: What, How, When, Why and Why Notcited by 28
  3. 2026HealthBench Professional: Evaluating Large Language Models on Real Clinician Chatscited by 26
  4. 2024TWIN-GPT: Digital Twins for Clinical Trials via Large Language Modelcited by 24
  5. 2024Formal Theorem Proving by Rewarding LLMs to Decompose Proofs Hierarchicallycited by 22
  6. 2025ER-Reason: A Benchmark Dataset for LLM Clinical Reasoning in the Emergency Roomcited by 22
  7. 2024Biomedical Visual Instruction Tuning with Clinician Preference Alignmentcited by 21
  8. 2024Large Language Model Benchmarks in Medical Taskscited by 17
  9. 2023Distilling Large Language Models for Matching Patients to Clinical Trialscited by 16
  10. 2024A Survey on Large Language Models from General Purpose to Medical Applications: Datasets, Methodologies, and Evaluationscited by 16
  11. 2024A dataset and benchmark for hospital course summarization with adapted large language modelscited by 11
  12. 2023Unifying Corroborative and Contributive Attributions in Large Language Modelscited by 10
  13. 2024DOLOMITES: Domain-Specific Long-Form Methodical Taskscited by 8
  14. 2024Fine-Tuning a Local LLaMA-3 Large Language Model for Automated Privacy-Preserving Physician Letter Generation in Radiation Oncologycited by 8
  15. 2025Can Large Language Models Match the Conclusions of Systematic Reviews?cited by 8
  16. 2023Question Answering for Electronic Health Records: A Scoping Review of datasets and modelscited by 7
  17. 2023"What's important here?": Opportunities and Challenges of Using LLMs in Retrieving Information from Web Interfacescited by 6
  18. 2024FactEHR: A Dataset for Evaluating Factuality in Clinical Notes Using LLMscited by 5
  19. 2024Fine-tuning Large Language Models with Limited Data: A Survey and Practical Guidecited by 4
  20. 2025Evaluating Retrieval-Augmented Generation vs. Long-Context Input for Clinical Reasoning over EHRscited by 4
  21. 2025Cross-Representation Benchmarking in Time-Series Electronic Health Records for Clinical Outcome Predictioncited by 4
  22. 2026Can LLM Agents Generate Real-World Evidence? Evaluating Observational Studies in Medical Databasescited by 4
  23. 2023Ascle: A Python Natural Language Processing Toolkit for Medical Text Generationcited by 3
  24. 2024Recent Advances, Applications, and Open Challenges in Machine Learning for Health: Reflections from Research Roundtables at ML4H 2023 Symposiumcited by 3
  25. 2025Conversation Forests: The Key to Fine Tuning Large Language Models for Multi-Turn Medical Conversations is Branchingcited by 2
  26. 2025Static Sandboxes Are Inadequate: Modeling Societal Complexity Requires Open-Ended Co-Evolution in LLM-Based Multi-Agent Simulationscited by 2
  27. 2026LLMs can construct powerful representations and streamline sample-efficient supervised learningcited by 2
  28. 2026From Token to Token Pair: Efficient Prompt Compression for Large Language Models in Clinical Predictioncited by 2
  29. 2026EHRBench: An Automated and Reliable EHR-based Benchmark for Clinical Decision Making with LLMscited by 2
  30. 202547B Mixture-of-Experts Beats 671B Dense Models on Chinese Medical Examinationscited by 1
  31. 2025Balancing Safety and Helpfulness in Healthcare AI Assistants through Iterative Preference Alignmentcited by 1
  32. 2026MedBeads: An AI-Native Clinical Context Graph Built from Immutable Beads and Reconstructable Clinical Linkscited by 1
  33. 2026RECON: Benchmarking Agent Memory for Compositional Reasoning over Long Contextscited by 1
  34. 2025Holistic Artificial Intelligence in Medicine; improved performance and explainabilitycited by 0
  35. 2025Automated Evaluation can Distinguish the Good and Bad AI Responses to Patient Questions about Hospitalizationcited by 0
  36. 2025Investigating LLM Capabilities on Long Context Comprehension for Medical Question Answeringcited by 0
  37. 2026CARE: Privacy-Compliant Agentic Reasoning with Evidence Discordancecited by 0
  38. 2026Ten Headache Specialists versus Artificial Intelligence for Clinical Literature Summarization: A Critical Evaluation and Comparisoncited by 0

50 most cited of 56 All 56 (JSON)