# Towards Understanding the Generalization of Medical Text-to-SQL Models and Datasets

arXiv: https://arxiv.org/abs/2303.12898 · HTML: https://mlsanity.com/paper/2303.12898/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 38 references that are in the map.
- First posted on arXiv 22 Mar 2023.

## Builds on (most cited first, 19 of 19)

- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task](https://mlsanity.com/paper/1809.08887/) (2018)
- [Improving Neural Machine Translation Models with Monolingual Data](https://mlsanity.com/paper/1511.06709/) (2015)
- [Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning](https://mlsanity.com/paper/1709.00103/) (2017)
- [RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parsers](https://mlsanity.com/paper/1911.04942/) (2019)
- [PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Models](https://mlsanity.com/paper/2109.05093/) (2021)
- [UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Models](https://mlsanity.com/paper/2201.05966/) (2022)
- [SQLNet: Generating Structured Queries From Natural Language Without Reinforcement Learning](https://mlsanity.com/paper/1711.04436/) (2017)
- [Evaluating the Text-to-SQL Capabilities of Large Language Models](https://mlsanity.com/paper/2204.00498/) (2022)
- [Learning a Neural Semantic Parser from User Feedback](https://mlsanity.com/paper/1704.08760/) (2017)
- [TypeSQL: Knowledge-based Type-Aware Neural Text-to-SQL Generation](https://mlsanity.com/paper/1804.09769/) (2018)
- [Text-to-SQL Generation for Question Answering on Electronic Medical Records](https://mlsanity.com/paper/1908.01839/) (2019)
- [Representing Schema Structure with Graph Neural Networks for Text-to-SQL Parsing](https://mlsanity.com/paper/1905.06241/) (2019)
- [Robust Text-to-SQL Generation with Execution-Guided Decoding](https://mlsanity.com/paper/1807.03100/) (2018)
- [Memory-efficient NLLB-200: Language-specific Expert Pruning of a Massively Multilingual Machine Translation Model](https://mlsanity.com/paper/2212.09811/) (2022)
- [Proton: Probing Schema Linking Information from Pre-trained Language Models for Text-to-SQL Parsing](https://mlsanity.com/paper/2206.14017/) (2022)
- [Weakly Supervised Text-to-SQL Parsing through Question Decomposition](https://mlsanity.com/paper/2112.06311/) (2021)
- [Dynamic Prediction of ICU Mortality Risk Using Domain Adaptation](https://mlsanity.com/paper/1912.10080/) (2019)

## Built on by (most cited first, 9 of 9)

- [EHRXQA: A Multi-Modal Question Answering Dataset for Electronic Health Records with Chest X-ray Images](https://mlsanity.com/paper/2310.18652/) (2023)
- [Retrieval augmented text-to-SQL generation for epidemiological question answering using electronic health records](https://mlsanity.com/paper/2403.09226/) (2024)
- [Question Answering for Electronic Health Records: A Scoping Review of datasets and models](https://mlsanity.com/paper/2310.08759/) (2023)
- [Unmasking Database Vulnerabilities: Zero-Knowledge Schema Inference Attacks in Text-to-SQL Systems](https://mlsanity.com/paper/2406.14545/) (2024)
- [Generating patient cohorts from electronic health records using two-step retrieval-augmented text-to-SQL generation](https://mlsanity.com/paper/2502.21107/) (2025)
- [KU-DMIS at EHRSQL 2024:Generating SQL query via question templatization in EHR](https://mlsanity.com/paper/2406.00014/) (2024)
- [Beyond Text-to-SQL for IoT Defense: A Comprehensive Framework for Querying and Classifying IoT Threats](https://mlsanity.com/paper/2406.17574/) (2024)
- [Bridging Global Intent with Local Details: A Hierarchical Representation Approach for Semantic Validation in Text-to-SQL](https://mlsanity.com/paper/2512.22744/) (2025)
- [Large Language Models for Structured Clinical Data Analysis: Dual-Agent Grounding and Validation](https://mlsanity.com/paper/2609.34039/) (2026)

## Abstract (arXiv)

Electronic medical records (EMRs) are stored in relational databases. It can be challenging to access the required information if the user is unfamiliar with the database schema or general database fundamentals. Hence, researchers have explored text-to-SQL generation methods that provide healthcare professionals direct access to EMR data without needing a database expert. However, currently available datasets have been essentially "solved" with state-of-the-art models achieving accuracy greater than or near 90%. In this paper, we show that there is still a long way to go before solving text-to-SQL generation in the medical domain. To show this, we create new splits of the existing medical text-to-SQL dataset MIMICSQL that better measure the generalizability of the resulting models. We evaluate state-of-the-art language models on our new split showing substantial drops in performance with accuracy dropping from up to 92% to 28%, thus showing substantial room for improvement. Moreover, we introduce a novel data augmentation approach to improve the generalizability of the language models. Overall, this paper is the first step towards developing more robust text-to-SQL models in the medical domain.footnoteThe dataset and code will be released upon acceptance.