# BenchPress: A Human-in-the-Loop Annotation System for Rapid Text-to-SQL Benchmark Curation

arXiv: https://arxiv.org/abs/2510.13853 · HTML: https://mlsanity.com/paper/2510.13853/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 28 references that are in the map.
- First posted on arXiv 11 Oct 2025.

## Builds on (most cited first, 15 of 15)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task](https://mlsanity.com/paper/1809.08887/) (2018)
- [Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs](https://mlsanity.com/paper/2305.03111/) (2023)
- [Semantic Evaluation for Text-to-SQL with Distilled Test Suites](https://mlsanity.com/paper/2010.02840/) (2020)
- [BEAVER: An Enterprise Benchmark for Text-to-SQL](https://mlsanity.com/paper/2409.02038/) (2024)
- [SQL-to-Text Generation with Graph-to-Sequence Model](https://mlsanity.com/paper/1809.05255/) (2018)
- [Automatic Metadata Extraction for Text-to-SQL](https://mlsanity.com/paper/2505.19988/) (2025)
- [L2CEval: Evaluating Language-to-Code Generation Capabilities of Large Language Models](https://mlsanity.com/paper/2309.17446/) (2023)
- [Table-To-Text generation and pre-training with TabT5](https://mlsanity.com/paper/2210.09162/) (2022)
- [SMiLE: Schema-augmented Multi-level Contrastive Learning for Knowledge Graph Link Prediction](https://mlsanity.com/paper/2210.04870/) (2022)
- [GenEdit: Compounding Operators and Continuous Improvement to Tackle Text-to-SQL in the Enterprise](https://mlsanity.com/paper/2503.21602/) (2025)
- [Using Causal Inference to Explore Government Policy Impact on Computer Usage](https://mlsanity.com/paper/2503.09957/) (2025)

## Built on by (most cited first, 3 of 3)

- [EGREFINE: An Execution-Grounded Optimization Framework for Text-to-SQL Schema Refinement](https://mlsanity.com/paper/2605.00628/) (2026)
- [SQLMorph: Query Mutation and Fine-Grained Metrics for Text-to-SQL Evaluation](https://mlsanity.com/paper/2609.08950/) (2026)
- [From Automation to Collaboration: Human-in-the-Loop Methods for Safe and Trustworthy NLP](https://mlsanity.com/paper/2605.25226/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have been successfully applied to many tasks, including text-to-SQL generation. However, much of this work has focused on publicly available datasets, such as Fiben, Spider, and Bird. Our earlier work showed that LLMs are much less effective in querying large private enterprise data warehouses and released Beaver, the first private enterprise text-to-SQL benchmark. To create Beaver, we leveraged SQL logs, which are often readily available. However, manually annotating these logs to identify which natural language questions they answer is a daunting task. Asking database administrators, who are highly trained experts, to take on additional work to construct and validate corresponding natural language utterances is not only challenging but also quite costly. To address this challenge, we introduce BenchPress, a human-in-the-loop system designed to accelerate the creation of domain-specific text-to-SQL benchmarks. Given a SQL query, BenchPress uses retrieval-augmented generation (RAG) and LLMs to propose multiple natural language descriptions. Human experts then select, rank, or edit these drafts to ensure accuracy and domain alignment. We evaluated BenchPress on annotated enterprise SQL logs, demonstrating that LLM-assisted annotation drastically reduces the time and effort required to create high-quality benchmarks. Our results show that combining human verification with LLM-generated suggestions enhances annotation accuracy, benchmark reliability, and model evaluation robustness. By streamlining the creation of custom benchmarks, BenchPress offers researchers and practitioners a mechanism for assessing text-to-SQL models on a given domain-specific workload. BenchPress is freely available via our public GitHub repository at https://github.com/fabian-wenz/enterprise-txt2sql and is also accessible on our website at http://dsg-mcgraw.csail.mit.edu:5000.