# SGSH: Stimulate Large Language Models with Skeleton Heuristics for Knowledge Base Question Generation

arXiv: https://arxiv.org/abs/2404.01923 · HTML: https://mlsanity.com/paper/2404.01923/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 37 references that are in the map.
- First posted on arXiv 2 Apr 2024.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [What Makes Good In-Context Examples for GPT-3?](https://mlsanity.com/paper/2101.06804/) (2021)
- [The Web as a Knowledge-base for Answering Complex Questions](https://mlsanity.com/paper/1803.06643/) (2018)
- [Open Domain Question Answering Using Early Fusion of Knowledge Bases and Text](https://mlsanity.com/paper/1809.00782/) (2018)
- [What In-Context Learning "Learns" In-Context: Disentangling Task Recognition and Task Learning](https://mlsanity.com/paper/2305.09731/) (2023)
- [Knowledge-Augmented Language Model Prompting for Zero-Shot Knowledge Graph Question Answering](https://mlsanity.com/paper/2306.04136/) (2023)
- [Improving Multi-hop Knowledge Base Question Answering by Learning Intermediate Supervision Signals](https://mlsanity.com/paper/2101.03737/) (2021)
- [JointGT: Graph-Text Joint Representation Learning for Text Generation from Knowledge Graphs](https://mlsanity.com/paper/2106.10502/) (2021)
- [An Interpretable Reasoning Network for Multi-Relation Question Answering](https://mlsanity.com/paper/1801.04726/) (2018)
- [Enhancing Few-shot Text-to-SQL Capabilities of Large Language Models: A Study on Prompt Design Strategies](https://mlsanity.com/paper/2305.12586/) (2023)
- [Improving ChatGPT Prompt for Code Generation](https://mlsanity.com/paper/2305.08360/) (2023)
- [Math Word Problem Generation with Mathematical Consistency and Problem Context Constraints](https://mlsanity.com/paper/2109.04546/) (2021)
- [Prompting Large Language Models with Chain-of-Thought for Few-Shot Knowledge Base Question Generation](https://mlsanity.com/paper/2310.08395/) (2023)
- [Knowledge-enriched, Type-constrained and Grammar-guided Question Generation over Knowledge Bases](https://mlsanity.com/paper/2010.03157/) (2020)
- [AutoQGS: Auto-Prompt for Low-Resource Knowledge-based Question Generation from SPARQL](https://mlsanity.com/paper/2208.12461/) (2022)
- [Generating Questions for Knowledge Bases via Incorporating Diversified Contexts and Answer-Aware Loss](https://mlsanity.com/paper/1910.13108/) (2019)
- [Diversifying Question Generation over Knowledge Base via External Natural Questions](https://mlsanity.com/paper/2309.14362/) (2023)

## Built on by (most cited first, 6 of 6)

- [GraphGen: Enhancing Supervised Fine-Tuning for LLMs with Knowledge-Driven Synthetic Data Generation](https://mlsanity.com/paper/2505.20416/) (2025)
- [PCQPR: Proactive Conversational Question Planning with Reflection](https://mlsanity.com/paper/2410.01363/) (2024)
- [A Survey of Ontology Expansion for Conversational Understanding](https://mlsanity.com/paper/2410.15019/) (2024)
- [FIQ: Fundamental Question Generation with the Integration of Question Embeddings for Video Question Answering](https://mlsanity.com/paper/2507.12816/) (2025)
- [Foundational Question Generation for Video Question Answering via an Embedding-Integrated Approach](https://mlsanity.com/paper/2511.17618/) (2025)
- [ChronoQG: Towards a Temporally Expressive and Hop-Bounded Benchmark for Temporal Knowledge Graph Question Generation](https://mlsanity.com/paper/2607.14770/) (2026)

## Abstract (arXiv)

Knowledge base question generation (KBQG) aims to generate natural language questions from a set of triplet facts extracted from KB. Existing methods have significantly boosted the performance of KBQG via pre-trained language models (PLMs) thanks to the richly endowed semantic knowledge. With the advance of pre-training techniques, large language models (LLMs) (e.g., GPT-3.5) undoubtedly possess much more semantic knowledge. Therefore, how to effectively organize and exploit the abundant knowledge for KBQG becomes the focus of our study. In this work, we propose SGSH--a simple and effective framework to Stimulate GPT-3.5 with Skeleton Heuristics to enhance KBQG. The framework incorporates "skeleton heuristics", which provides more fine-grained guidance associated with each input to stimulate LLMs to generate optimal questions, encompassing essential elements like the question phrase and the auxiliary verb.More specifically, we devise an automatic data construction strategy leveraging ChatGPT to construct a skeleton training dataset, based on which we employ a soft prompting approach to train a BART model dedicated to generating the skeleton associated with each input. Subsequently, skeleton heuristics are encoded into the prompt to incentivize GPT-3.5 to generate desired questions. Extensive experiments demonstrate that SGSH derives the new state-of-the-art performance on the KBQG tasks.