# DB-GPT-Hub: Towards Open Benchmarking Text-to-SQL Empowered by Large Language Models

arXiv: https://arxiv.org/abs/2406.11434 · HTML: https://mlsanity.com/paper/2406.11434/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Jun 2024.

## Built on by (most cited first, 6 of 6)

- [A Survey on Employing Large Language Models for Text-to-SQL Tasks](https://mlsanity.com/paper/2407.15186/) (2024)
- [LookBench: A Live and Holistic Open Benchmark for Fashion Image Retrieval](https://mlsanity.com/paper/2601.14706/) (2026)
- [Exploring the Landscape of Text-to-SQL with Large Language Models: Progresses, Challenges and Opportunities](https://mlsanity.com/paper/2505.23838/) (2025)
- [ROMAS: A Role-Based Multi-Agent System for Database monitoring and Planning](https://mlsanity.com/paper/2412.13520/) (2024)
- [DORAEMON: A Unified Library for Visual Object Modeling and Representation Learning at Scale](https://mlsanity.com/paper/2511.04394/) (2025)
- [Beyond Retrieval: A Multitask Benchmark and Model for Code Search](https://mlsanity.com/paper/2605.04615/) (2026)

## Abstract (arXiv)

Large language models (LLMs) becomes the dominant paradigm for the challenging task of text-to-SQL. LLM-empowered text-to-SQL methods are typically categorized into prompting-based and tuning approaches. Compared to prompting-based methods, benchmarking fine-tuned LLMs for text-to-SQL is important yet under-explored, partially attributed to the prohibitively high computational cost. In this paper, we present DB-GPT-Hub, an open benchmark suite for LLM-empowered text-to-SQL, which primarily focuses on tuning LLMs at large scales. The proposed benchmark consists of: 1. a standardized and comprehensive evaluation of text-to-SQL tasks by fine-tuning medium to large-sized open LLMs; 2. a modularized and easy-to-extend codebase with mainstream LLMs and experimental scenarios supported, which prioritizes fine-tuning methods but can be easily extended to prompt-based setting. Our work investigates the potential gains and the performance boundaries of tuning approaches, compared to prompting approaches and explores optimal solutions tailored to specific scenarios. We hope DB-GPT-Hub, along with these findings, enables further research and broad applications that would otherwise be difficult owing to the absence of a dedicated open benchmark. The project code has been released at https://github.com/eosphoros-ai/DB-GPT-Hub.