# Leveraging LLMs in Scholarly Knowledge Graph Question Answering

arXiv: https://arxiv.org/abs/2311.09841 · HTML: https://mlsanity.com/paper/2311.09841/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 18 references that are in the map.
- First posted on arXiv 16 Nov 2023.

## Builds on (most cited first, 9 of 9)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Out of One, Many: Using Language Models to Simulate Human Samples](https://mlsanity.com/paper/2209.06899/) (2022)
- [Large Language Models are few(1)-shot Table Reasoners](https://mlsanity.com/paper/2210.06710/) (2022)
- [Evaluating Open-Domain Question Answering in the Era of Large Language Models](https://mlsanity.com/paper/2305.06984/) (2023)
- [DBLP-QuAD: A Question Answering Dataset over the DBLP Scholarly Knowledge Graph](https://mlsanity.com/paper/2303.13351/) (2023)
- [Question Answering on Scholarly Knowledge Graphs](https://mlsanity.com/paper/2006.01527/) (2020)
- [GETT-QA: Graph Embedding based T2T Transformer for Knowledge Graph Question Answering](https://mlsanity.com/paper/2303.13284/) (2023)

## Built on by (most cited first, 13 of 13)

- [Are Large Language Models Good Fact Checkers: A Preliminary Study](https://mlsanity.com/paper/2311.17355/) (2023)
- [CG-RAG: Research Question Answering by Citation Graph Retrieval-Augmented LLMs](https://mlsanity.com/paper/2501.15067/) (2025)
- [SPARQL Generation with Entity Pre-trained GPT for KG Question Answering](https://mlsanity.com/paper/2402.00969/) (2024)
- [Way to Specialist: Closing Loop Between Specialized LLM and Evolving Domain Knowledge Graph](https://mlsanity.com/paper/2411.19064/) (2024)
- [FIRESPARQL: A LLM-based Framework for SPARQL Query Generation over Scholarly Knowledge Graphs](https://mlsanity.com/paper/2508.10467/) (2025)
- [ARUQULA -- An LLM based Text2SPARQL Approach using ReAct and Knowledge Graph Exploration Utilities](https://mlsanity.com/paper/2510.02200/) (2025)
- [Scholarly Question Answering using Large Language Models in the NFDI4DataScience Gateway](https://mlsanity.com/paper/2406.07257/) (2024)
- [Contri(e)ve: Context + Retrieve for Scholarly Question Answering](https://mlsanity.com/paper/2409.09010/) (2024)
- [KERL: Knowledge-Enhanced Personalized Recipe Recommendation using Large Language Models](https://mlsanity.com/paper/2505.14629/) (2025)
- [Enhancing Manufacturing Knowledge Access with LLMs and Context-aware Prompting](https://mlsanity.com/paper/2507.22619/) (2025)
- [PKG API: A Tool for Personal Knowledge Graph Management](https://mlsanity.com/paper/2402.07540/) (2024)
- [How do Scaling Laws Apply to Knowledge Graph Engineering Tasks? The Impact of Model Size on Large Language Model Performance](https://mlsanity.com/paper/2505.16276/) (2025)
- [Reading Between the Citations: A Typed Claim Network for Scientific Literature](https://mlsanity.com/paper/2605.30966/) (2026)

## Abstract (arXiv)

This paper presents a scholarly Knowledge Graph Question Answering (KGQA) that answers bibliographic natural language questions by leveraging a large language model (LLM) in a few-shot manner. The model initially identifies the top-n similar training questions related to a given test question via a BERT-based sentence encoder and retrieves their corresponding SPARQL. Using the top-n similar question-SPARQL pairs as an example and the test question creates a prompt. Then pass the prompt to the LLM and generate a SPARQL. Finally, runs the SPARQL against the underlying KG - ORKG (Open Research KG) endpoint and returns an answer. Our system achieves an F1 score of 99.0%, on SciQA - one of the Scholarly-QALD-23 challenge benchmarks.