# SM3-Text-to-Query: Synthetic Multi-Model Medical Text-to-Query Benchmark

arXiv: https://arxiv.org/abs/2411.05521 · HTML: https://mlsanity.com/paper/2411.05521/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 49 references that are in the map.
- First posted on arXiv 8 Nov 2024.

## Builds on (most cited first, 16 of 16)

- [Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task](https://mlsanity.com/paper/1809.08887/) (2018)
- [Large Language Models as Optimizers](https://mlsanity.com/paper/2309.03409/) (2023)
- [Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs](https://mlsanity.com/paper/2305.03111/) (2023)
- [Improving Text-to-SQL Evaluation Methodology](https://mlsanity.com/paper/1806.09029/) (2018)
- [KaggleDBQA: Realistic Evaluation of Text-to-SQL Parsers](https://mlsanity.com/paper/2106.11455/) (2021)
- [Text-to-SQL Generation for Question Answering on Electronic Medical Records](https://mlsanity.com/paper/1908.01839/) (2019)
- [EHRXQA: A Multi-Modal Question Answering Dataset for Electronic Health Records with Chest X-ray Images](https://mlsanity.com/paper/2310.18652/) (2023)
- [EHRSQL: A Practical Text-to-SQL Benchmark for Electronic Health Records](https://mlsanity.com/paper/2301.07695/) (2023)
- [ScienceBenchmark: A Complex Real-World Benchmark for Evaluating Natural Language to SQL Systems](https://mlsanity.com/paper/2306.04743/) (2023)
- [Is Stack Overflow Obsolete? An Empirical Study of the Characteristics of ChatGPT Answers to Stack Overflow Questions](https://mlsanity.com/paper/2308.02312/) (2023)
- [Knowledge Graph-based Question Answering with Electronic Health Records](https://mlsanity.com/paper/2010.09394/) (2020)
- [Evaluating the Data Model Robustness of Text-to-SQL Systems Based on Real User Queries](https://mlsanity.com/paper/2402.08349/) (2024)
- [EPI-SQL: Enhancing Text-to-SQL Translation with Error-Prevention Instructions](https://mlsanity.com/paper/2404.14453/) (2024)
- [StatBot.Swiss: Bilingual Open Data Exploration in Natural Language](https://mlsanity.com/paper/2406.03170/) (2024)
- [Spider4SPARQL: A Complex Benchmark for Evaluating Knowledge Graph Question Answering Systems](https://mlsanity.com/paper/2309.16248/) (2023)
- [Interactive Ontology Matching with Cost-Efficient Learning](https://mlsanity.com/paper/2404.07663/) (2024)

## Built on by (most cited first, 14 of 14)

- [SHARE: An SLM-based Hierarchical Action CorREction Assistant for Text-to-SQL](https://mlsanity.com/paper/2506.00391/) (2025)
- [U2-BENCH: Benchmarking Large Vision-Language Models on Ultrasound Understanding](https://mlsanity.com/paper/2505.17779/) (2025)
- [Multi-Modal Data Exploration via Language Agents](https://mlsanity.com/paper/2412.18428/) (2024)
- [BiomedSQL: Text-to-SQL for Scientific Reasoning on Biomedical Knowledge Bases](https://mlsanity.com/paper/2505.20321/) (2025)
- [Bench360: Benchmarking Local LLM Inference from 360 Degrees](https://mlsanity.com/paper/2511.16682/) (2025)
- [Patient-Similarity Cohort Reasoning in Clinical Text-to-SQL](https://mlsanity.com/paper/2601.09876/) (2026)
- [HealthBranches: Synthesizing Clinically-Grounded Question Answering Datasets via Decision Pathways](https://mlsanity.com/paper/2508.07308/) (2025)
- [ClinQueryAgent: A Conversational Agent for Population Health Management](https://mlsanity.com/paper/2605.18768/) (2026)
- [RAS: Reflection-Augmented Scaling with In-Context Learning for Executable Cypher Query Generation](https://mlsanity.com/paper/2605.22937/) (2026)
- [Applying Language Models in Clinical Medicine: Recent Trends and Perspectives](https://mlsanity.com/paper/2609.34780/) (2026)
- [CypherTurn: A Multi-Turn Benchmark for Conversational Text-to-Cypher Evaluation and the Autonomy Divergence](https://mlsanity.com/paper/2609.36987/) (2026)
- [AptMQL-Bench: From Text-to-SQL to Text-to-MQL via Access-Pattern Schema Design and Data-Preserving Migration](https://mlsanity.com/paper/2610.02770/) (2026)
- [From Benchmarks to Production: A Text-to-SQL System for Complex Financial Data](https://mlsanity.com/paper/2610.03524/) (2026)
- [ROAR: Unifying Runs across Heterogeneous AI-Driven Research Systems](https://mlsanity.com/paper/2610.03966/) (2026)

## Abstract (arXiv)

Electronic health records (EHRs) are stored in various database systems with different database models on heterogeneous storage architectures, such as relational databases, document stores, or graph databases. These different database models have a big impact on query complexity and performance. While this has been a known fact in database research, its implications for the growing number of Text-to-Query systems have surprisingly not been investigated so far. In this paper, we present SM3-Text-to-Query, the first multi-model medical Text-to-Query benchmark based on synthetic patient data from Synthea, following the SNOMED-CT taxonomy -- a widely used knowledge graph ontology covering medical terminology. SM3-Text-to-Query provides data representations for relational databases (PostgreSQL), document stores (MongoDB), and graph databases (Neo4j and GraphDB (RDF)), allowing the evaluation across four popular query languages, namely SQL, MQL, Cypher, and SPARQL. We systematically and manually develop 408 template questions, which we augment to construct a benchmark of 10K diverse natural language question/query pairs for these four query languages (40K pairs overall). On our dataset, we evaluate several common in-context-learning (ICL) approaches for a set of representative closed and open-source LLMs. Our evaluation sheds light on the trade-offs between database models and query languages for different ICL strategies and LLMs. Last, SM3-Text-to-Query is easily extendable to additional query languages or real, standard-based patient databases.