BIS: NL2SQL Service Evaluation Benchmark for Business Intelligence Scenarios
Not on the map. Why?
Abstract (arXiv)
NL2SQL (Natural Language to Structured Query Language) transformation has seen wide adoption in Business Intelligence (BI) applications in recent years. However, existing NL2SQL benchmarks are not suitable for production BI scenarios, as they are not designed for common business intelligence questions. To address this gap, we have developed a new benchmark focused on typical NL questions in industrial BI scenarios. We discuss the challenges of constructing a BI-focused benchmark and the shortcomings of existing benchmarks. Additionally, we introduce question categories in our benchmark that reflect common BI inquiries. Lastly, we propose two novel semantic similarity evaluation metrics for assessing NL2SQL capabilities in BI applications and services.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 6 of its 13 references that are in the map. First posted on arXiv 30 Oct 2024.
Builds on
6 of 13 references.
References · 6
- 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,120
- 2017Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learningcited by 679
- 2018Improving Text-to-SQL Evaluation Methodologycited by 133
- 2022A Survey on Text-to-SQL Parsing: Concepts, Methods, and Future Directionscited by 51
- 2019A Pilot Study for Chinese SQL Semantic Parsingcited by 33
- 2022Deep Learning Driven Natural Languages Text to SQL Query Conversion: A Surveycited by 11
Built on by
Citing papers per year: 2025: 3.
3 papers.