# BEAVER: An Enterprise Benchmark for Text-to-SQL

arXiv: https://arxiv.org/abs/2409.02038 · HTML: https://mlsanity.com/paper/2409.02038/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 41 references that are in the map.
- First posted on arXiv 3 Sep 2024.

## Builds on (most cited first, 20 of 20)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task](https://mlsanity.com/paper/1809.08887/) (2018)
- [Towards General Text Embeddings with Multi-stage Contrastive Learning](https://mlsanity.com/paper/2308.03281/) (2023)
- [Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs](https://mlsanity.com/paper/2305.03111/) (2023)
- [DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correction](https://mlsanity.com/paper/2304.11015/) (2023)
- [Large Language Models Struggle to Learn Long-Tail Knowledge](https://mlsanity.com/paper/2211.08411/) (2022)
- [Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation](https://mlsanity.com/paper/2308.15363/) (2023)
- [RESDSQL: Decoupling Schema Linking and Skeleton Parsing for Text-to-SQL](https://mlsanity.com/paper/2302.05965/) (2023)
- [CHESS: Contextual Harnessing for Efficient SQL Synthesis](https://mlsanity.com/paper/2405.16755/) (2024)
- [KaggleDBQA: Realistic Evaluation of Text-to-SQL Parsers](https://mlsanity.com/paper/2106.11455/) (2021)
- [AnglE-optimized Text Embeddings](https://mlsanity.com/paper/2309.12871/) (2023)
- [ScienceBenchmark: A Complex Real-World Benchmark for Evaluating Natural Language to SQL Systems](https://mlsanity.com/paper/2306.04743/) (2023)
- [MultiTabQA: Generating Tabular Answers for Multi-Table Question Answering](https://mlsanity.com/paper/2305.12820/) (2023)
- [Is Table Retrieval a Solved Problem? Exploring Join-Aware Multi-Table Retrieval](https://mlsanity.com/paper/2404.09889/) (2024)
- [UNITE: A Unified Benchmark for Text-to-SQL Evaluation](https://mlsanity.com/paper/2305.16265/) (2023)

## Built on by (most cited first, 30 of 42)

- [Is Long Context All You Need? Leveraging LLM's Extended Context for NL2SQL](https://mlsanity.com/paper/2501.12372/) (2025)
- [SWE-SQL: Illuminating LLM Pathways to Solve User SQL Issues in Real-World Applications](https://mlsanity.com/paper/2506.18951/) (2025)
- [EnrichIndex: Using LLMs to Enrich Retrieval Indices Offline](https://mlsanity.com/paper/2504.03598/) (2025)
- [Think2SQL: Reinforce LLM Reasoning Capabilities for Text2SQL](https://mlsanity.com/paper/2504.15077/) (2025)
- [EllieSQL: Cost-Efficient Text-to-SQL with Complexity-Aware Routing](https://mlsanity.com/paper/2503.22402/) (2025)
- [Pervasive Annotation Errors Break Text-to-SQL Benchmarks and Leaderboards](https://mlsanity.com/paper/2601.08778/) (2026)
- [An LLM-Based Approach for Insight Generation in Data Analysis](https://mlsanity.com/paper/2503.11664/) (2025)
- [Can we Retrieve Everything All at Once? ARM: An Alignment-Oriented LLM-based Retrieval Method](https://mlsanity.com/paper/2501.18539/) (2025)
- [DataLab: A Unified Platform for LLM-Powered Business Intelligence](https://mlsanity.com/paper/2412.02205/) (2024)
- [GenEdit: Compounding Operators and Continuous Improvement to Tackle Text-to-SQL in the Enterprise](https://mlsanity.com/paper/2503.21602/) (2025)
- [Exploring the Landscape of Text-to-SQL with Large Language Models: Progresses, Challenges and Opportunities](https://mlsanity.com/paper/2505.23838/) (2025)
- [RubikSQL: Lifelong Learning Agentic Knowledge Base as an Industrial NL2SQL System](https://mlsanity.com/paper/2508.17590/) (2025)
- [BenchPress: A Human-in-the-Loop Annotation System for Rapid Text-to-SQL Benchmark Curation](https://mlsanity.com/paper/2510.13853/) (2025)
- [SQaLe: A Large Text-to-SQL Corpus Grounded in Real Schemas](https://mlsanity.com/paper/2602.22223/) (2025)
- [Semantic Layers for Reliable LLM-Powered Data Analytics: A Paired Benchmark of Accuracy and Hallucination Across Three Frontier Models](https://mlsanity.com/paper/2604.25149/) (2026)
- [Evaluating NL2SQL via SQL2NL](https://mlsanity.com/paper/2509.04657/) (2025)
- [Exploring Multi-Table Retrieval Through Iterative Search](https://mlsanity.com/paper/2511.13418/) (2025)
- [CORE-T: COherent REtrieval of Tables for Text-to-SQL](https://mlsanity.com/paper/2601.13111/) (2026)
- [Fine-Grained Table Retrieval Through the Lens of Complex Queries](https://mlsanity.com/paper/2603.07146/) (2026)
- [EGREFINE: An Execution-Grounded Optimization Framework for Text-to-SQL Schema Refinement](https://mlsanity.com/paper/2605.00628/) (2026)
- [Never the Number: Structural Abstention for AI Systems Whose Answers Are Consumed as Fact](https://mlsanity.com/paper/2608.13926/) (2026)
- [SQLMorph: Query Mutation and Fine-Grained Metrics for Text-to-SQL Evaluation](https://mlsanity.com/paper/2609.08950/) (2026)
- [Semantic Caching and Intent-Driven Context Optimization for Multi-Agent Natural Language to Code Systems](https://mlsanity.com/paper/2601.11687/) (2026)
- [GraphER: An Efficient Graph-Based Enrichment and Reranking Method for Retrieval-Augmented Generation](https://mlsanity.com/paper/2603.24925/) (2026)
- [A Demonstration of SQLyzr: A Platform for Fine-Grained Text-to-SQL Evaluation and Analysis](https://mlsanity.com/paper/2604.21214/) (2026)
- [Retrieve Only Relevant Tables Whether Few or Many: Adaptive Table Retrieval Method](https://mlsanity.com/paper/2605.18766/) (2026)
- [BADGER: Bridging Agentic and Deterministic Evaluation for Generative Enterprise Reasoning](https://mlsanity.com/paper/2606.02109/) (2026)
- [SOMA-SQL: Resolving Multi-Source Ambiguity in NL-to-SQL via Synthetic Log and Execution Probing](https://mlsanity.com/paper/2606.11424/) (2026)
- [Spider 2.0-AIFunc: Extending Real-World Text-to-SQL to AI-Native SQL Workflows](https://mlsanity.com/paper/2607.06229/) (2026)
- [Finding the Right Tables and Columns: A Benchmark and Corpus-Adaptive Embeddings for SQL Schema Retrieval](https://mlsanity.com/paper/2607.13311/) (2026)

## Abstract (arXiv)

Existing text-to-SQL benchmarks have largely been constructed from public databases with well-structured schemas and simplistic question-SQL pairs. While large language models (LLMs) excel on these settings, their efficacy in complex private enterprise environments, characterized by intricate schemas, domain knowledge, and analytical user queries involving sophisticated structures and functions, remains unproven. To bridge this gap, we introduce BEAVER, the first text-to-SQL benchmark derived from private data warehouses. It comprises 9128 question-SQL pairs sourced from real-world query logs and 812 tables across 19 diverse domains. Building this benchmark is challenging because (1) enterprise query logs are scarce due to privacy constraints, and (2) existing all-or-nothing evaluation metrics based on accuracy make error diagnosis difficult -- especially when producing a correct query involves solving multiple compounded challenges, such as domain knowledge and query complexity. We address these issues at two levels. At the dataset level, we synthesize high-fidelity, expert-verified queries that increase dataset size and isolate individual challenges or combine them, producing queries focused on domain knowledge, query complexity, and both. At the evaluation level, we provide human annotations and evaluation metrics for five critical subtasks to enable fine-grained analysis. Our evaluation reveals a significant performance gap compared to existing benchmarks: SOTA agentic frameworks using the advanced model GPT-5.2 achieve only 10.8% accuracy. When provided with all subtask annotations as oracle hints, accuracy increases to 30.1%, confirming that a major bottleneck lies in correctly resolving these subtasks. Finally, we provide a taxonomy of the residual errors that persist even with subtask hints, identifying specific challenges such as the use of advanced functions.