# The Role of Accuracy and Validation Effectiveness in Conversational Business Analytics

arXiv: https://arxiv.org/abs/2411.12128 · HTML: https://mlsanity.com/paper/2411.12128/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 26 of its 96 references that are in the map.
- First posted on arXiv 18 Nov 2024.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task](https://mlsanity.com/paper/1809.08887/) (2018)
- [Large Language Models](https://mlsanity.com/paper/2307.05782/) (2023)
- [Extending Context Window of Large Language Models via Positional Interpolation](https://mlsanity.com/paper/2306.15595/) (2023)
- [Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs](https://mlsanity.com/paper/2305.03111/) (2023)
- [Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning](https://mlsanity.com/paper/1709.00103/) (2017)
- [DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correction](https://mlsanity.com/paper/2304.11015/) (2023)
- [CoSQL: A Conversational Text-to-SQL Challenge Towards Cross-Domain Natural Language Interfaces to Databases](https://mlsanity.com/paper/1909.05378/) (2019)
- [What Do We Want From Explainable Artificial Intelligence (XAI)? -- A Stakeholder Perspective on XAI and a Conceptual Model Guiding Interdisciplinary XAI Research](https://mlsanity.com/paper/2102.07817/) (2021)
- [Chat2VIS: Generating Data Visualisations via Natural Language using ChatGPT, Codex and GPT-3 Large Language Models](https://mlsanity.com/paper/2302.02094/) (2023)
- [Speak to your Parser: Interactive Text-to-SQL with Natural Language Feedback](https://mlsanity.com/paper/2005.02539/) (2020)
- [Model-based Interactive Semantic Parsing: A Unified Framework and A Text-to-SQL Case Study](https://mlsanity.com/paper/1910.05389/) (2019)
- [MAGIC: Generating Self-Correction Guideline for In-Context Text-to-SQL](https://mlsanity.com/paper/2406.12692/) (2024)
- [Large Language Models as Data Preprocessors](https://mlsanity.com/paper/2308.16361/) (2023)
- [Automated Data Visualization from Natural Language via Large Language Models: An Exploratory Study](https://mlsanity.com/paper/2404.17136/) (2024)
- [Large Language Models Streamline Automated Machine Learning for Clinical Studies](https://mlsanity.com/paper/2308.14120/) (2023)
- [Retrieval-augmented GPT-3.5-based Text-to-SQL Framework with Sample-aware Prompting and Dynamic Revision Chain](https://mlsanity.com/paper/2307.05074/) (2023)
- [Dubo-SQL: Diverse Retrieval-Augmented Generation and Fine Tuning for Text-to-SQL](https://mlsanity.com/paper/2404.12560/) (2024)
- [SQLucid: Grounding Natural Language Database Queries with Interactive Explanations](https://mlsanity.com/paper/2409.06178/) (2024)
- [Automatic Data Transformation Using Large Language Model: An Experimental Study on Building Energy Data](https://mlsanity.com/paper/2309.01957/) (2023)
- [An Empirical Validation of Cognitive Complexity as a Measure of Source Code Understandability](https://mlsanity.com/paper/2007.12520/) (2020)
- [QueryVis: Logic-based diagrams help users understand complicated SQL queries faster](https://mlsanity.com/paper/2004.11375/) (2020)

## Abstract (arXiv)

This study examines conversational business analytics, an approach that utilizes AI to address the technical competency gaps that hinder end users from effectively using traditional self-service analytics. By facilitating natural language interactions, conversational business analytics aims to empower end users to independently retrieve data and generate insights. The analysis focuses on Text-to-SQL as a representative technology for translating natural language requests into SQL statements. Developing theoretical models grounded in expected utility theory, this study identifies the conditions under which conversational business analytics, through partial or full support, can outperform delegation to human experts. The results indicate that partial support, focusing solely on information generation by AI, is viable when the accuracy of AI-generated SQL queries leads to a profit that surpasses the performance of a human expert. In contrast, full support includes not only information generation but also validation through explanations provided by the AI, and requires sufficiently high validation effectiveness to be reliable. However, user-based validation presents challenges, such as misjudgment and rejection of valid SQL queries, which may limit the effectiveness of conversational business analytics. These challenges underscore the need for robust validation mechanisms, including improved user support, automated processes, and methods for assessing quality independent of the technical competency of end users.