# Both Ends Count! Just How Good are LLM Agents at "Text-to-Big SQL"?

arXiv: https://arxiv.org/abs/2602.21480 · HTML: https://mlsanity.com/paper/2602.21480/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use.
- Builds on 26 of its 55 references that are in the map.
- First posted on arXiv 25 Feb 2026.

## Placed in

- LLM agents and tool use / [Web, search, and research agents](https://mlsanity.com/agents/web-search-and-research-agents/)

## Builds on (most cited first, 26 of 26)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task](https://mlsanity.com/paper/1809.08887/) (2018)
- [Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs](https://mlsanity.com/paper/2305.03111/) (2023)
- [PaperBench: Evaluating AI's Ability to Replicate AI Research](https://mlsanity.com/paper/2504.01848/) (2025)
- [Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows](https://mlsanity.com/paper/2411.07763/) (2024)
- [AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges](https://mlsanity.com/paper/2505.10468/) (2025)
- [Next-Generation Database Interfaces: A Survey of LLM-based Text-to-SQL](https://mlsanity.com/paper/2406.08426/) (2024)
- [OmniSQL: Synthesizing High-quality Text-to-SQL Data at Scale](https://mlsanity.com/paper/2503.02240/) (2025)
- [The Dawn of Natural Language to SQL: Are We Fully Ready?](https://mlsanity.com/paper/2406.01265/) (2024)
- [Tool Documentation Enables Zero-Shot Tool-Usage with Large Language Models](https://mlsanity.com/paper/2308.00675/) (2023)
- [Barbarians at the Gate: How AI is Upending Systems Research](https://mlsanity.com/paper/2510.06189/) (2025)
- [ReFoRCE: A Text-to-SQL Agent with Self-Refinement, Consensus Enforcement, and Column Exploration](https://mlsanity.com/paper/2502.00675/) (2025)
- [FinSQL: Model-Agnostic LLMs-based Text-to-SQL Framework for Financial Analysis](https://mlsanity.com/paper/2401.10506/) (2024)
- [Alpha-SQL: Zero-Shot Text-to-SQL using Monte Carlo Tree Search](https://mlsanity.com/paper/2502.17248/) (2025)
- [Supporting Our AI Overlords: Redesigning Data Systems to be Agent-First](https://mlsanity.com/paper/2509.00997/) (2025)
- [Is Long Context All You Need? Leveraging LLM's Extended Context for NL2SQL](https://mlsanity.com/paper/2501.12372/) (2025)
- [Large Language Model Enhanced Text-to-SQL Generation: A Survey](https://mlsanity.com/paper/2410.06011/) (2024)
- [MAG-SQL: Multi-Agent Generative Approach with Soft Schema Linking and Iterative Sub-SQL Refinement for Text-to-SQL](https://mlsanity.com/paper/2408.07930/) (2024)
- [Extracting books from production language models](https://mlsanity.com/paper/2601.02671/) (2026)
- [Understanding, Detecting, and Repairing Real-World In-Context-Learning-Based Text-to-SQL Errors](https://mlsanity.com/paper/2501.09310/) (2025)
- [Abacus: A Cost-Based Optimizer for Semantic Operator Systems](https://mlsanity.com/paper/2505.14661/) (2025)
- [SQLfuse: Enhancing Text-to-SQL Performance through Comprehensive LLM Synergy](https://mlsanity.com/paper/2407.14568/) (2024)
- [Beyond Relational: Semantic-Aware Multi-Modal Analytics with LLM-Native Query Optimization](https://mlsanity.com/paper/2511.19830/) (2025)
- [PalimpChat: Declarative and Interactive AI analytics](https://mlsanity.com/paper/2502.03368/) (2025)
- [Broken Chains: The Cost of Incomplete Reasoning in LLMs](https://mlsanity.com/paper/2602.14444/) (2026)
- [Unified Tool Integration for LLMs: A Protocol-Agnostic Approach to Function Calling](https://mlsanity.com/paper/2508.02979/) (2025)

## Built on by (most cited first, 1 of 1)

- [BudgetSchemaBench: A Budget-Swept Diagnostic for Schema Context in Text-to-SQL](https://mlsanity.com/paper/2610.00092/) (2026)

## Abstract (arXiv)

Text-to-SQL and Big Data are both extensively benchmarked fields, yet there is limited research that evaluates them jointly. In the real world, Text-to-SQL systems are often embedded with Big Data workflows, such as large-scale data processing or interactive data analytics. We refer to this as ``Text-to-Big SQL''. However, existing text-to-SQL benchmarks remain narrowly scoped and overlook the cost and performance implications that arise at scale. For instance, translation errors that are minor on small datasets lead to substantial cost and latency overheads as data scales, a relevant issue completely ignored by text-to-SQL metrics. In this paper, we overcome this overlooked challenge by introducing novel and representative metrics for evaluating Text-to-Big SQL. Our study focuses on production-level LLM agents, a database-agnostic system adaptable to diverse user needs. Via an extensive evaluation of frontier models, we show that text-to-SQL metrics are insufficient for Big Data. In contrast, our proposed text-to-Big SQL metrics accurately reflect execution efficiency, cost, and the impact of data scale. For example, GPT-4o compensates for roughly 7% lower accuracy than the top-performing later-generation models with up to a 12.16x speedup, while GPT-5.2 is more than twice as cost-effective as Gemini 3 Pro at large input scales.