# A Benchmark to Understand the Role of Knowledge Graphs on Large Language Model's Accuracy for Question Answering on Enterprise SQL Databases

arXiv: https://arxiv.org/abs/2311.07509 · HTML: https://mlsanity.com/paper/2311.07509/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 24 references that are in the map.
- First posted on arXiv 13 Nov 2023.

## Builds on (most cited first, 8 of 8)

- [Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task](https://mlsanity.com/paper/1809.08887/) (2018)
- [Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning](https://mlsanity.com/paper/1709.00103/) (2017)
- [Unifying Large Language Models and Knowledge Graphs: A Roadmap](https://mlsanity.com/paper/2306.08302/) (2023)
- [Holistic Evaluation of Text-To-Image Models](https://mlsanity.com/paper/2311.04287/) (2023)
- [Improving Text-to-SQL Evaluation Methodology](https://mlsanity.com/paper/1806.09029/) (2018)
- [KaggleDBQA: Realistic Evaluation of Text-to-SQL Parsers](https://mlsanity.com/paper/2106.11455/) (2021)
- [Learning a Neural Semantic Parser from User Feedback](https://mlsanity.com/paper/1704.08760/) (2017)
- [Towards Foundation Models for Knowledge Graph Reasoning](https://mlsanity.com/paper/2310.04562/) (2023)

## Built on by (most cited first, 25 of 25)

- [Knowledge Graphs as Context Sources for LLM-Based Explanations of Learning Recommendations](https://mlsanity.com/paper/2403.03008/) (2024)
- [Large Language Models Meet Knowledge Graphs for Question Answering: Synthesis and Opportunities](https://mlsanity.com/paper/2505.20099/) (2025)
- [Is Long Context All You Need? Leveraging LLM's Extended Context for NL2SQL](https://mlsanity.com/paper/2501.12372/) (2025)
- [Increasing the LLM Accuracy for Question Answering: Ontologies to the Rescue!](https://mlsanity.com/paper/2405.11706/) (2024)
- [From Natural Language to SQL: Review of LLM-based Text-to-SQL Systems](https://mlsanity.com/paper/2410.01066/) (2024)
- [Graph RAG-Tool Fusion](https://mlsanity.com/paper/2502.07223/) (2025)
- [Supporting Student Decisions on Learning Recommendations: An LLM-Based Chatbot with Knowledge Graph Contextualization for Conversational Explainability and Mentoring](https://mlsanity.com/paper/2401.08517/) (2024)
- [CommunityKG-RAG: Leveraging Community Structures in Knowledge Graphs for Advanced Retrieval-Augmented Generation in Fact-Checking](https://mlsanity.com/paper/2408.08535/) (2024)
- [Hybrid LLM/Rule-based Approaches to Business Insights Generation from Structured Data](https://mlsanity.com/paper/2404.15604/) (2024)
- [SpotIt: Evaluating Text-to-SQL Evaluation with Formal Verification](https://mlsanity.com/paper/2510.26840/) (2025)
- [DeepMEL: A Multi-Agent Collaboration Framework for Multimodal Entity Linking](https://mlsanity.com/paper/2508.15876/) (2025)
- [Semantic Layers for Reliable LLM-Powered Data Analytics: A Paired Benchmark of Accuracy and Hallucination Across Three Frontier Models](https://mlsanity.com/paper/2604.25149/) (2026)
- [TIC: Translate-Infer-Compile for accurate "text to plan" using LLMs and Logical Representations](https://mlsanity.com/paper/2402.06608/) (2024)
- [Use of a Structured Knowledge Base Enhances Metadata Curation by Large Language Models](https://mlsanity.com/paper/2404.05893/) (2024)
- [CSR-RAG: An Efficient Retrieval System for Text-to-SQL on the Enterprise Scale](https://mlsanity.com/paper/2601.06564/) (2026)
- [Lucy: Think and Reason to Solve Text-to-SQL](https://mlsanity.com/paper/2407.05153/) (2024)
- [Using off-the-shelf LLMs to query enterprise data by progressively revealing ontologies](https://mlsanity.com/paper/2410.09244/) (2024)
- [HI-SQL: Optimizing Text-to-SQL Systems through Dynamic Hint Integration](https://mlsanity.com/paper/2506.18916/) (2025)
- [Knowledge Conceptualization Impacts RAG Efficacy](https://mlsanity.com/paper/2507.09389/) (2025)
- [Natural Language Access to Domain-Specific Metadata: A Reusable Framework for LLM Query Generation](https://mlsanity.com/paper/2607.18029/) (2026)
- [Bounded Semantic Planning and Deterministic Compilation for Reliable Enterprise Text-to-SQL](https://mlsanity.com/paper/2608.16663/) (2026)
- [Walking on the DARKSIDE](https://mlsanity.com/paper/2608.23370/) (2026)
- [Beyond Vector Search: Comparing Classical RAG with Hybrid GraphRAG for Climate Science Q&A](https://mlsanity.com/paper/2608.28766/) (2026)
- [Natural Language Knowledge Graph Query Execution: Leveraging Controlled Semantics in the LLM Context Window](https://mlsanity.com/paper/2609.14652/) (2026)
- [Which Part of the Context Layer Does the Work? Separating Semantic Content from Retrieval Scaffolding in Text-to-SQL Agents](https://mlsanity.com/paper/2609.22259/) (2026)

## Abstract (arXiv)

Enterprise applications of Large Language Models (LLMs) hold promise for question answering on enterprise SQL databases. However, the extent to which LLMs can accurately respond to enterprise questions in such databases remains unclear, given the absence of suitable Text-to-SQL benchmarks tailored to enterprise settings. Additionally, the potential of Knowledge Graphs (KGs) to enhance LLM-based question answering by providing business context is not well understood. This study aims to evaluate the accuracy of LLM-powered question answering systems in the context of enterprise questions and SQL databases, while also exploring the role of knowledge graphs in improving accuracy. To achieve this, we introduce a benchmark comprising an enterprise SQL schema in the insurance domain, a range of enterprise queries encompassing reporting to metrics, and a contextual layer incorporating an ontology and mappings that define a knowledge graph. Our primary finding reveals that question answering using GPT-4, with zero-shot prompts directly on SQL databases, achieves an accuracy of 16%. Notably, this accuracy increases to 54% when questions are posed over a Knowledge Graph representation of the enterprise SQL database. Therefore, investing in Knowledge Graph provides higher accuracy for LLM powered question answering systems.