mlsanityData to 7 Oct 2026

EHRSQL: A Practical Text-to-SQL Benchmark for Electronic Health Records

arXiv 2301.07695 ↗First posted 16 Jan 2023cs.CL cs.AIcited by 42

Not on the map. Why?

Abstract (arXiv)

We present a new text-to-SQL dataset for electronic health records (EHRs). The utterances were collected from 222 hospital staff members, including physicians, nurses, and insurance review and health records teams. To construct the QA dataset on structured EHR data, we conducted a poll at a university hospital and used the responses to create seed questions. We then manually linked these questions to two open-source EHR databases, MIMIC-III and eICU, and included various time expressions and held-out unanswerable questions in the dataset, which were also collected from the poll. Our dataset poses a unique set of challenges: the model needs to 1) generate SQL queries that reflect a wide range of needs in the hospital, including simple retrieval and complex operations such as calculating survival rate, 2) understand various time expressions to answer time-sensitive questions in healthcare, and 3) distinguish whether a given question is answerable or unanswerable. We believe our dataset, EHRSQL, can serve as a practical benchmark for developing and assessing QA models on structured EHR data and take a step further towards bridging the gap between text-to-SQL research and its real-life deployment in healthcare. EHRSQL is available at https://github.com/glee4810/EHRSQL.

Key facts. Cited by 42 papers in the map (data through 7 Oct 2026). Builds on 26 of its 47 references that are in the map. First posted on arXiv 16 Jan 2023.

Builds on

26 of 47 references.

References · 26

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  4. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  5. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  6. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,120
  7. 2017Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learningcited by 679
  8. 2017Adversarial Examples for Evaluating Reading Comprehension Systemscited by 568
  9. 2020Beyond English-Centric Multilingual Machine Translationcited by 559
  10. 2020Multilingual Translation with Extensible Multilingual Pretraining and Finetuningcited by 287
  11. 2021On Hallucination and Predictive Uncertainty in Conditional Language Generationcited by 176
  12. 2019SParC: Cross-Domain Semantic Parsing in Contextcited by 141
14 more
  1. 2018emrQA: A Large Corpus for Question Answering on Electronic Medical Recordscited by 134
  2. 2018Improving Text-to-SQL Evaluation Methodologycited by 133
  3. 2017Simple and Effective Multi-Paragraph Reading Comprehensioncited by 120
  4. 2021KaggleDBQA: Realistic Evaluation of Text-to-SQL Parserscited by 105
  5. 2019Text-to-SQL Generation for Question Answering on Electronic Medical Recordscited by 76
  6. 2020Learning Contextual Representations for Semantic Parsing with Generation-Augmented Pre-Trainingcited by 65
  7. 2019Out-of-domain Detection for Natural Language Understanding in Dialog Systemscited by 62
  8. 2020Speak to your Parser: Interactive Text-to-SQL with Natural Language Feedbackcited by 44
  9. 2021Text-to-SQL in the Wild: A Naturally-Occurring Dataset Based on Stack Exchange Datacited by 43
  10. 2020Knowledge Graph-based Question Answering with Electronic Health Recordscited by 19
  11. 2022DrugEHRQA: A Question Answering Dataset on Structured and Unstructured Electronic Health Records For Medicine Related Queriescited by 17
  12. 2020Did You Ask a Good Question? A Cross-Domain Question Intention Classification Benchmark for Text-to-SQLcited by 16
  13. 2021OodGAN: Generative Adversarial Network for Out-of-Domain Data Generationcited by 15
  14. 2021Question Answering for Complex Electronic Health Records Database using Unified Encoder-Decoder Architecturecited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 7; 2025: 15; 2026: 16.

42 papers.

Sort:

By area:

  1. 2023Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLscited by 667
  2. 2024EHRAgent: Code Empowers Large Language Models for Few-shot Complex Tabular Reasoning on Electronic Health Recordscited by 75
  3. 2023EHRXQA: A Multi-Modal Question Answering Dataset for Electronic Health Records with Chest X-ray Imagescited by 53
  4. 2023ECG-QA: A Comprehensive Question Answering Dataset Combined With Electrocardiogramcited by 52
  5. 2025MedAgentGym: A Scalable Agentic Training Environment for Code-Centric Reasoning in Biomedical Data Sciencecited by 22
  6. 2025DataFlow: An LLM-Driven Framework for Unified Data Preparation and Workflow Automation in the Era of Data-Centric AIcited by 22
  7. 2025FHIR-AgentBench: Benchmarking LLM Agents for Realistic Interoperable EHR Question Answeringcited by 20
  8. 2024TrustSQL: Benchmarking Text-to-SQL Reliability with Penalty-Based Scoringcited by 15
  9. 2025Text2SQL-Flow: A Robust SQL-Aware Data Augmentation Framework for Text-to-SQLcited by 11
  10. 2024SM3-Text-to-Query: Synthetic Multi-Model Medical Text-to-Query Benchmarkcited by 10
  11. 2024Almanac Copilot: Towards Autonomous Electronic Health Record Navigationcited by 9
  12. 2024A Survey of Large Language Model-Based Generative AI for Text-to-SQL: Benchmarks, Applications, Use Cases, and Challengescited by 9
30 more
  1. 2023Question Answering for Electronic Health Records: A Scoping Review of datasets and modelscited by 7
  2. 2026LongMedBench: Benchmarking Medical Agents for Long-Horizon Clinical Decision-Makingcited by 6
  3. 2025Generating patient cohorts from electronic health records using two-step retrieval-augmented text-to-SQL generationcited by 4
  4. 2025BiomedSQL: Text-to-SQL for Scientific Reasoning on Biomedical Knowledge Basescited by 4
  5. 2025Exploring the Landscape of Text-to-SQL with Large Language Models: Progresses, Challenges and Opportunitiescited by 4
  6. 2024Towards Unbiased Evaluation of Detecting Unanswerable Questions in EHRSQLcited by 3
  7. 2025M3: Conversational LLMs Simplify Secure Clinical Data Access, Understanding, and Analysiscited by 3
  8. 2025SQaLe: A Large Text-to-SQL Corpus Grounded in Real Schemascited by 3
  9. 2024A Systematic Review of NeurIPS Dataset Management Practicescited by 2
  10. 2026An Efficient and Effective Evaluator for Text2SQL Models on Unseen and Unlabeled Datacited by 2
  11. 2026Can LLM Agents Generate Real-World Evidence? Evaluating Observational Studies in Medical Databasescited by 2
  12. 2026CodeClinic: Evaluating Automation of Coding Skills for Clinical Reasoning Agentscited by 2
  13. 2025MedInsightBench: Evaluating Medical Analytics Agents Through Multi-Step Insight Discovery in Multimodal Medical Datacited by 1
  14. 2025Mitigating hallucinations in healthcare LLMs with granular fact-checking and domain-specific adaptationcited by 1
  15. 2026EHRNavigator: A Multi-Agent System for Patient-Level Clinical Question Answering over Heterogeneous Electronic Health Recordscited by 1
  16. 2026From Natural Language to Executable Option Strategies via Large Language Modelscited by 1
  17. 2025EMR-AGENT: Automating Cohort and Feature Extraction from EMR Databasescited by 0
  18. 2025Reliable Curation of EHR Dataset via Large Language Models under Environmental Constraintscited by 0
  19. 2025SCARE: A Benchmark for SQL Correction and Question Answerability Classification for Reliable EHR Question Answeringcited by 0
  20. 2025Bridging Global Intent with Local Details: A Hierarchical Representation Approach for Semantic Validation in Text-to-SQLcited by 0
  21. 2026RA-QA: A Benchmarking System for Respiratory Audio Question Answering Under Real-World Heterogeneitycited by 0
  22. 2026FHIRPath-QA: Executable Question Answering over FHIR Electronic Health Recordscited by 0
  23. 2026ESL-Bench: An Event-Driven Synthetic Longitudinal Benchmark for Health Agentscited by 0
  24. 2026SemanticAgent: A Semantics-Aware Framework for Text-to-SQL Data Synthesiscited by 0
  25. 2026FastOMOP: A Foundational Architecture for Reliable Agentic Real-World Evidence Generation on OMOP CDM datacited by 0
  26. 2026Anatomy of a Query: W5H Dimensions and FAR Patterns for Text-to-SQL Evaluationcited by 0
  27. 2026Knowledge Distillation for Low-Resource Open-source Text-to-SQL Modelcited by 0
  28. 2026The Verbose Context Problem in Medical Recordscited by 0
  29. 2026DRL: A Deterministic Relational Middleware Layer for Transaction-Safe Enterprise NL2SQL Under Schema-Graph Scalingcited by 0
  30. 2026Synthetic Hospital: An Open, Verifiable, Physician-Validated Longitudinal EHR Benchmarkcited by 0