mlsanityData to 14 Oct 2026

A Benchmark to Understand the Role of Knowledge Graphs on Large Language Model's Accuracy for Question Answering on Enterprise SQL Databases

arXiv 2311.07509 ↗First posted 13 Nov 2023cs.AI cs.CL cs.DBcited by 25

Not on the map. Why?

Abstract (arXiv)

Enterprise applications of Large Language Models (LLMs) hold promise for question answering on enterprise SQL databases. However, the extent to which LLMs can accurately respond to enterprise questions in such databases remains unclear, given the absence of suitable Text-to-SQL benchmarks tailored to enterprise settings. Additionally, the potential of Knowledge Graphs (KGs) to enhance LLM-based question answering by providing business context is not well understood. This study aims to evaluate the accuracy of LLM-powered question answering systems in the context of enterprise questions and SQL databases, while also exploring the role of knowledge graphs in improving accuracy. To achieve this, we introduce a benchmark comprising an enterprise SQL schema in the insurance domain, a range of enterprise queries encompassing reporting to metrics, and a contextual layer incorporating an ontology and mappings that define a knowledge graph. Our primary finding reveals that question answering using GPT-4, with zero-shot prompts directly on SQL databases, achieves an accuracy of 16%. Notably, this accuracy increases to 54% when questions are posed over a Knowledge Graph representation of the enterprise SQL database. Therefore, investing in Knowledge Graph provides higher accuracy for LLM powered question answering systems.

Key facts. Cited by 25 papers in the map (data through 14 Oct 2026). Builds on 8 of its 24 references that are in the map. First posted on arXiv 13 Nov 2023.

Builds on

8 of 24 references.

References · 8

  1. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,139
  2. 2017Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learningcited by 687
  3. 2023Unifying Large Language Models and Knowledge Graphs: A Roadmapcited by 668
  4. 2023Holistic Evaluation of Text-To-Image Modelscited by 160
  5. 2018Improving Text-to-SQL Evaluation Methodologycited by 135
  6. 2021KaggleDBQA: Realistic Evaluation of Text-to-SQL Parserscited by 106
  7. 2017Learning a Neural Semantic Parser from User Feedbackcited by 99
  8. 2023Towards Foundation Models for Knowledge Graph Reasoningcited by 86

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 10; 2025: 7; 2026: 8.

25 papers.

Sort:

By area:

  1. 2024Knowledge Graphs as Context Sources for LLM-Based Explanations of Learning Recommendationscited by 31
  2. 2025Large Language Models Meet Knowledge Graphs for Question Answering: Synthesis and Opportunitiescited by 31
  3. 2025Is Long Context All You Need? Leveraging LLM's Extended Context for NL2SQLcited by 30
  4. 2024Increasing the LLM Accuracy for Question Answering: Ontologies to the Rescue!cited by 18
  5. 2024From Natural Language to SQL: Review of LLM-based Text-to-SQL Systemscited by 16
  6. 2025Graph RAG-Tool Fusioncited by 15
  7. 2024Supporting Student Decisions on Learning Recommendations: An LLM-Based Chatbot with Knowledge Graph Contextualization for Conversational Explainability and Mentoringcited by 11
  8. 2024CommunityKG-RAG: Leveraging Community Structures in Knowledge Graphs for Advanced Retrieval-Augmented Generation in Fact-Checkingcited by 11
  9. 2024Hybrid LLM/Rule-based Approaches to Business Insights Generation from Structured Datacited by 7
  10. 2025SpotIt: Evaluating Text-to-SQL Evaluation with Formal Verificationcited by 4
  11. 2025DeepMEL: A Multi-Agent Collaboration Framework for Multimodal Entity Linkingcited by 3
  12. 2026Semantic Layers for Reliable LLM-Powered Data Analytics: A Paired Benchmark of Accuracy and Hallucination Across Three Frontier Modelscited by 3
13 more
  1. 2024TIC: Translate-Infer-Compile for accurate "text to plan" using LLMs and Logical Representationscited by 2
  2. 2024Use of a Structured Knowledge Base Enhances Metadata Curation by Large Language Modelscited by 2
  3. 2026CSR-RAG: An Efficient Retrieval System for Text-to-SQL on the Enterprise Scalecited by 2
  4. 2024Lucy: Think and Reason to Solve Text-to-SQLcited by 0
  5. 2024Using off-the-shelf LLMs to query enterprise data by progressively revealing ontologiescited by 0
  6. 2025HI-SQL: Optimizing Text-to-SQL Systems through Dynamic Hint Integrationcited by 0
  7. 2025Knowledge Conceptualization Impacts RAG Efficacycited by 0
  8. 2026Natural Language Access to Domain-Specific Metadata: A Reusable Framework for LLM Query Generationcited by 0
  9. 2026Bounded Semantic Planning and Deterministic Compilation for Reliable Enterprise Text-to-SQLcited by 0
  10. 2026Walking on the DARKSIDEcited by 0
  11. 2026Beyond Vector Search: Comparing Classical RAG with Hybrid GraphRAG for Climate Science Q&Acited by 0
  12. 2026Natural Language Knowledge Graph Query Execution: Leveraging Controlled Semantics in the LLM Context Windowcited by 0
  13. 2026Which Part of the Context Layer Does the Work? Separating Semantic Content from Retrieval Scaffolding in Text-to-SQL Agentscited by 0