# BIS: NL2SQL Service Evaluation Benchmark for Business Intelligence Scenarios

arXiv: https://arxiv.org/abs/2410.22925 · HTML: https://mlsanity.com/paper/2410.22925/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 13 references that are in the map.
- First posted on arXiv 30 Oct 2024.

## Builds on (most cited first, 6 of 6)

- [Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task](https://mlsanity.com/paper/1809.08887/) (2018)
- [Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning](https://mlsanity.com/paper/1709.00103/) (2017)
- [Improving Text-to-SQL Evaluation Methodology](https://mlsanity.com/paper/1806.09029/) (2018)
- [A Survey on Text-to-SQL Parsing: Concepts, Methods, and Future Directions](https://mlsanity.com/paper/2208.13629/) (2022)
- [A Pilot Study for Chinese SQL Semantic Parsing](https://mlsanity.com/paper/1909.13293/) (2019)
- [Deep Learning Driven Natural Languages Text to SQL Query Conversion: A Survey](https://mlsanity.com/paper/2208.04415/) (2022)

## Built on by (most cited first, 3 of 3)

- [RubikSQL: Lifelong Learning Agentic Knowledge Base as an Industrial NL2SQL System](https://mlsanity.com/paper/2508.17590/) (2025)
- [Formalising Software Requirements using Large Language Models](https://mlsanity.com/paper/2506.10704/) (2025)
- [Working Document -- Formalising Software Requirements with Large Language Models](https://mlsanity.com/paper/2506.14627/) (2025)

## Abstract (arXiv)

NL2SQL (Natural Language to Structured Query Language) transformation has seen wide adoption in Business Intelligence (BI) applications in recent years. However, existing NL2SQL benchmarks are not suitable for production BI scenarios, as they are not designed for common business intelligence questions. To address this gap, we have developed a new benchmark focused on typical NL questions in industrial BI scenarios. We discuss the challenges of constructing a BI-focused benchmark and the shortcomings of existing benchmarks. Additionally, we introduce question categories in our benchmark that reflect common BI inquiries. Lastly, we propose two novel semantic similarity evaluation metrics for assessing NL2SQL capabilities in BI applications and services.