# Evaluating Large Language Models in Semantic Parsing for Conversational Question Answering over Knowledge Graphs

arXiv: https://arxiv.org/abs/2401.01711 · HTML: https://mlsanity.com/paper/2401.01711/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 24 references that are in the map.
- First posted on arXiv 3 Jan 2024.

## Builds on (most cited first, 18 of 18)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Survey of Hallucination in Natural Language Generation](https://mlsanity.com/paper/2202.03629/) (2022)
- [RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parsers](https://mlsanity.com/paper/1911.04942/) (2019)
- [Towards Complex Text-to-SQL in Cross-Domain Database with Intermediate Representation](https://mlsanity.com/paper/1905.08205/) (2019)
- [Beyond I.I.D.: Three Levels of Generalization for Question Answering on Knowledge Bases](https://mlsanity.com/paper/2011.07743/) (2020)
- [Editing-Based SQL Query Generation for Cross-Domain Context-Dependent Questions](https://mlsanity.com/paper/1909.00786/) (2019)
- [Complex Sequential Question Answering: Towards Learning to Converse Over Linked Question Answer Pairs with a Knowledge Graph](https://mlsanity.com/paper/1801.10314/) (2018)
- [How Far are We from Effective Context Modeling? An Exploratory Study on Semantic Parsing in Context](https://mlsanity.com/paper/2002.00652/) (2020)
- [Conversational Question Answering over Knowledge Graphs with Transformer and Graph Attention Networks](https://mlsanity.com/paper/2104.01569/) (2021)
- [Learning to Map Context-Dependent Sentences to Executable Formal Queries](https://mlsanity.com/paper/1804.06868/) (2018)
- [Analysing Mixed Initiatives and Search Strategies during Conversational Search](https://mlsanity.com/paper/2109.05955/) (2021)
- [Context Dependent Semantic Parsing: A Survey](https://mlsanity.com/paper/2011.00797/) (2020)
- [Semantic Parsing for Conversational Question Answering over Knowledge Graphs](https://mlsanity.com/paper/2301.12217/) (2023)
- [Investigating Conversational Search Behavior For Domain Exploration](https://mlsanity.com/paper/2301.04098/) (2023)

## Built on by (most cited first, 7 of 7)

- [A DbC Inspired Neurosymbolic Layer for Trustworthy Agent Design](https://mlsanity.com/paper/2508.03665/) (2025)
- [Doc2Spec: Synthesizing Formal Programming Specifications from Natural Language via Grammar Induction](https://mlsanity.com/paper/2602.04892/) (2026)
- [A Survey of the State-of-the-Art in Conversational Question Answering Systems](https://mlsanity.com/paper/2509.05716/) (2025)
- [SEAL: Self-Evolving Agentic Learning for Conversational Question Answering over Knowledge Graphs](https://mlsanity.com/paper/2512.04868/) (2025)
- [PerkwE_COQA: Enhanced Persian Conversational Question Answering by combining contextual keyword extraction with Large Language Models](https://mlsanity.com/paper/2404.05406/) (2024)
- [MetaExplainer: A Framework to Generate Multi-Type User-Centered Explanations for AI Systems](https://mlsanity.com/paper/2508.00300/) (2025)
- [OmniRetrieval: Unified Retrieval across Heterogeneous Knowledge Sources](https://mlsanity.com/paper/2605.29250/) (2026)

## Abstract (arXiv)

Conversational question answering systems often rely on semantic parsing to enable interactive information retrieval, which involves the generation of structured database queries from a natural language input. For information-seeking conversations about facts stored within a knowledge graph, dialogue utterances are transformed into graph queries in a process that is called knowledge-based conversational question answering. This paper evaluates the performance of large language models that have not been explicitly pre-trained on this task. Through a series of experiments on an extensive benchmark dataset, we compare models of varying sizes with different prompting techniques and identify common issue types in the generated output. Our results demonstrate that large language models are capable of generating graph queries from dialogues, with significant improvements achievable through few-shot prompting and fine-tuning techniques, especially for smaller models that exhibit lower zero-shot performance.