# EHRSQL: A Practical Text-to-SQL Benchmark for Electronic Health Records

arXiv: https://arxiv.org/abs/2301.07695 · HTML: https://mlsanity.com/paper/2301.07695/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 47 references that are in the map.
- First posted on arXiv 16 Jan 2023.

## Builds on (most cited first, 26 of 26)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [Know What You Don't Know: Unanswerable Questions for SQuAD](https://mlsanity.com/paper/1806.03822/) (2018)
- [Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task](https://mlsanity.com/paper/1809.08887/) (2018)
- [Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning](https://mlsanity.com/paper/1709.00103/) (2017)
- [Adversarial Examples for Evaluating Reading Comprehension Systems](https://mlsanity.com/paper/1707.07328/) (2017)
- [Beyond English-Centric Multilingual Machine Translation](https://mlsanity.com/paper/2010.11125/) (2020)
- [Multilingual Translation with Extensible Multilingual Pretraining and Finetuning](https://mlsanity.com/paper/2008.00401/) (2020)
- [On Hallucination and Predictive Uncertainty in Conditional Language Generation](https://mlsanity.com/paper/2103.15025/) (2021)
- [SParC: Cross-Domain Semantic Parsing in Context](https://mlsanity.com/paper/1906.02285/) (2019)
- [Improving Text-to-SQL Evaluation Methodology](https://mlsanity.com/paper/1806.09029/) (2018)
- [emrQA: A Large Corpus for Question Answering on Electronic Medical Records](https://mlsanity.com/paper/1809.00732/) (2018)
- [Simple and Effective Multi-Paragraph Reading Comprehension](https://mlsanity.com/paper/1710.10723/) (2017)
- [KaggleDBQA: Realistic Evaluation of Text-to-SQL Parsers](https://mlsanity.com/paper/2106.11455/) (2021)
- [Text-to-SQL Generation for Question Answering on Electronic Medical Records](https://mlsanity.com/paper/1908.01839/) (2019)
- [Learning Contextual Representations for Semantic Parsing with Generation-Augmented Pre-Training](https://mlsanity.com/paper/2012.10309/) (2020)
- [Out-of-domain Detection for Natural Language Understanding in Dialog Systems](https://mlsanity.com/paper/1909.03862/) (2019)
- [Speak to your Parser: Interactive Text-to-SQL with Natural Language Feedback](https://mlsanity.com/paper/2005.02539/) (2020)
- [Text-to-SQL in the Wild: A Naturally-Occurring Dataset Based on Stack Exchange Data](https://mlsanity.com/paper/2106.05006/) (2021)
- [Knowledge Graph-based Question Answering with Electronic Health Records](https://mlsanity.com/paper/2010.09394/) (2020)
- [DrugEHRQA: A Question Answering Dataset on Structured and Unstructured Electronic Health Records For Medicine Related Queries](https://mlsanity.com/paper/2205.01290/) (2022)
- [Did You Ask a Good Question? A Cross-Domain Question Intention Classification Benchmark for Text-to-SQL](https://mlsanity.com/paper/2010.12634/) (2020)
- [OodGAN: Generative Adversarial Network for Out-of-Domain Data Generation](https://mlsanity.com/paper/2104.02484/) (2021)
- [Question Answering for Complex Electronic Health Records Database using Unified Encoder-Decoder Architecture](https://mlsanity.com/paper/2111.14703/) (2021)

## Built on by (most cited first, 30 of 45)

- [Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs](https://mlsanity.com/paper/2305.03111/) (2023)
- [EHRAgent: Code Empowers Large Language Models for Few-shot Complex Tabular Reasoning on Electronic Health Records](https://mlsanity.com/paper/2401.07128/) (2024)
- [ECG-QA: A Comprehensive Question Answering Dataset Combined With Electrocardiogram](https://mlsanity.com/paper/2306.15681/) (2023)
- [EHRXQA: A Multi-Modal Question Answering Dataset for Electronic Health Records with Chest X-ray Images](https://mlsanity.com/paper/2310.18652/) (2023)
- [MedAgentGym: A Scalable Agentic Training Environment for Code-Centric Reasoning in Biomedical Data Science](https://mlsanity.com/paper/2506.04405/) (2025)
- [DataFlow: An LLM-Driven Framework for Unified Data Preparation and Workflow Automation in the Era of Data-Centric AI](https://mlsanity.com/paper/2512.16676/) (2025)
- [FHIR-AgentBench: Benchmarking LLM Agents for Realistic Interoperable EHR Question Answering](https://mlsanity.com/paper/2509.19319/) (2025)
- [TrustSQL: Benchmarking Text-to-SQL Reliability with Penalty-Based Scoring](https://mlsanity.com/paper/2403.15879/) (2024)
- [SM3-Text-to-Query: Synthetic Multi-Model Medical Text-to-Query Benchmark](https://mlsanity.com/paper/2411.05521/) (2024)
- [Text2SQL-Flow: A Robust SQL-Aware Data Augmentation Framework for Text-to-SQL](https://mlsanity.com/paper/2511.10192/) (2025)
- [A Survey of Large Language Model-Based Generative AI for Text-to-SQL: Benchmarks, Applications, Use Cases, and Challenges](https://mlsanity.com/paper/2412.05208/) (2024)
- [Almanac Copilot: Towards Autonomous Electronic Health Record Navigation](https://mlsanity.com/paper/2405.07896/) (2024)
- [CHI-Bench: Can AI Agents Automate End-to-End, Long-Horizon, Policy-Rich Healthcare Workflows?](https://mlsanity.com/paper/2605.16679/) (2026)
- [Question Answering for Electronic Health Records: A Scoping Review of datasets and models](https://mlsanity.com/paper/2310.08759/) (2023)
- [LongMedBench: Benchmarking Medical Agents for Long-Horizon Clinical Decision-Making](https://mlsanity.com/paper/2607.09322/) (2026)
- [Generating patient cohorts from electronic health records using two-step retrieval-augmented text-to-SQL generation](https://mlsanity.com/paper/2502.21107/) (2025)
- [BiomedSQL: Text-to-SQL for Scientific Reasoning on Biomedical Knowledge Bases](https://mlsanity.com/paper/2505.20321/) (2025)
- [Exploring the Landscape of Text-to-SQL with Large Language Models: Progresses, Challenges and Opportunities](https://mlsanity.com/paper/2505.23838/) (2025)
- [Can LLM Agents Generate Real-World Evidence? Evaluating Observational Studies in Medical Databases](https://mlsanity.com/paper/2603.22767/) (2026)
- [Towards Unbiased Evaluation of Detecting Unanswerable Questions in EHRSQL](https://mlsanity.com/paper/2405.01588/) (2024)
- [M3: Conversational LLMs Simplify Secure Clinical Data Access, Understanding, and Analysis](https://mlsanity.com/paper/2507.01053/) (2025)
- [SQaLe: A Large Text-to-SQL Corpus Grounded in Real Schemas](https://mlsanity.com/paper/2602.22223/) (2025)
- [A Systematic Review of NeurIPS Dataset Management Practices](https://mlsanity.com/paper/2411.00266/) (2024)
- [MedInsightBench: Evaluating Medical Analytics Agents Through Multi-Step Insight Discovery in Multimodal Medical Data](https://mlsanity.com/paper/2512.13297/) (2025)
- [An Efficient and Effective Evaluator for Text2SQL Models on Unseen and Unlabeled Data](https://mlsanity.com/paper/2603.07841/) (2026)
- [CodeClinic: Evaluating Automation of Coding Skills for Clinical Reasoning Agents](https://mlsanity.com/paper/2605.09675/) (2026)
- [Mitigating hallucinations in healthcare LLMs with granular fact-checking and domain-specific adaptation](https://mlsanity.com/paper/2512.16189/) (2025)
- [EHRNavigator: A Multi-Agent System for Patient-Level Clinical Question Answering over Heterogeneous Electronic Health Records](https://mlsanity.com/paper/2601.10020/) (2026)
- [From Natural Language to Executable Option Strategies via Large Language Models](https://mlsanity.com/paper/2603.16434/) (2026)
- [EMR-AGENT: Automating Cohort and Feature Extraction from EMR Databases](https://mlsanity.com/paper/2510.00549/) (2025)

## Abstract (arXiv)

We present a new text-to-SQL dataset for electronic health records (EHRs). The utterances were collected from 222 hospital staff members, including physicians, nurses, and insurance review and health records teams. To construct the QA dataset on structured EHR data, we conducted a poll at a university hospital and used the responses to create seed questions. We then manually linked these questions to two open-source EHR databases, MIMIC-III and eICU, and included various time expressions and held-out unanswerable questions in the dataset, which were also collected from the poll. Our dataset poses a unique set of challenges: the model needs to 1) generate SQL queries that reflect a wide range of needs in the hospital, including simple retrieval and complex operations such as calculating survival rate, 2) understand various time expressions to answer time-sensitive questions in healthcare, and 3) distinguish whether a given question is answerable or unanswerable. We believe our dataset, EHRSQL, can serve as a practical benchmark for developing and assessing QA models on structured EHR data and take a step further towards bridging the gap between text-to-SQL research and its real-life deployment in healthcare. EHRSQL is available at https://github.com/glee4810/EHRSQL.