mlsanityData to 7 Oct 2026

SM3-Text-to-Query: Synthetic Multi-Model Medical Text-to-Query Benchmark

arXiv 2411.05521 ↗First posted 8 Nov 2024cs.DB cs.AIcited by 10

Not on the map. Why?

Abstract (arXiv)

Electronic health records (EHRs) are stored in various database systems with different database models on heterogeneous storage architectures, such as relational databases, document stores, or graph databases. These different database models have a big impact on query complexity and performance. While this has been a known fact in database research, its implications for the growing number of Text-to-Query systems have surprisingly not been investigated so far. In this paper, we present SM3-Text-to-Query, the first multi-model medical Text-to-Query benchmark based on synthetic patient data from Synthea, following the SNOMED-CT taxonomy -- a widely used knowledge graph ontology covering medical terminology. SM3-Text-to-Query provides data representations for relational databases (PostgreSQL), document stores (MongoDB), and graph databases (Neo4j and GraphDB (RDF)), allowing the evaluation across four popular query languages, namely SQL, MQL, Cypher, and SPARQL. We systematically and manually develop 408 template questions, which we augment to construct a benchmark of 10K diverse natural language question/query pairs for these four query languages (40K pairs overall). On our dataset, we evaluate several common in-context-learning (ICL) approaches for a set of representative closed and open-source LLMs. Our evaluation sheds light on the trade-offs between database models and query languages for different ICL strategies and LLMs. Last, SM3-Text-to-Query is easily extendable to additional query languages or real, standard-based patient databases.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 16 of its 49 references that are in the map. First posted on arXiv 8 Nov 2024.

Builds on

16 of 49 references.

References · 16

  1. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,120
  2. 2023Large Language Models as Optimizerscited by 830
  3. 2023Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLscited by 667
  4. 2018Improving Text-to-SQL Evaluation Methodologycited by 133
  5. 2021KaggleDBQA: Realistic Evaluation of Text-to-SQL Parserscited by 105
  6. 2019Text-to-SQL Generation for Question Answering on Electronic Medical Recordscited by 76
  7. 2023EHRXQA: A Multi-Modal Question Answering Dataset for Electronic Health Records with Chest X-ray Imagescited by 53
  8. 2023EHRSQL: A Practical Text-to-SQL Benchmark for Electronic Health Recordscited by 42
  9. 2023ScienceBenchmark: A Complex Real-World Benchmark for Evaluating Natural Language to SQL Systemscited by 42
  10. 2023Is Stack Overflow Obsolete? An Empirical Study of the Characteristics of ChatGPT Answers to Stack Overflow Questionscited by 42
  11. 2020Knowledge Graph-based Question Answering with Electronic Health Recordscited by 19
  12. 2024Evaluating the Data Model Robustness of Text-to-SQL Systems Based on Real User Queriescited by 12
4 more
  1. 2024EPI-SQL: Enhancing Text-to-SQL Translation with Error-Prevention Instructionscited by 8
  2. 2024StatBot.Swiss: Bilingual Open Data Exploration in Natural Languagecited by 6
  3. 2023Spider4SPARQL: A Complex Benchmark for Evaluating Knowledge Graph Question Answering Systemscited by 3
  4. 2024Interactive Ontology Matching with Cost-Efficient Learningcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 5; 2026: 4.

10 papers.

  1. 2025SHARE: An SLM-based Hierarchical Action CorREction Assistant for Text-to-SQLcited by 15
  2. 2025U2-BENCH: Benchmarking Large Vision-Language Models on Ultrasound Understandingcited by 8
  3. 2024Multi-Modal Data Exploration via Language Agentscited by 7
  4. 2025BiomedSQL: Text-to-SQL for Scientific Reasoning on Biomedical Knowledge Basescited by 4
  5. 2025Bench360: Benchmarking Local LLM Inference from 360 Degreescited by 4
  6. 2026Patient-Similarity Cohort Reasoning in Clinical Text-to-SQLcited by 3
  7. 2025HealthBranches: Synthesizing Clinically-Grounded Question Answering Datasets via Decision Pathwayscited by 1
  8. 2026ClinQueryAgent: A Conversational Agent for Population Health Managementcited by 0
  9. 2026Applying Language Models in Clinical Medicine: Recent Trends and Perspectivescited by 0
  10. 2026CypherTurn: A Multi-Turn Benchmark for Conversational Text-to-Cypher Evaluation and the Autonomy Divergencecited by 0