# FlexKBQA: A Flexible LLM-Powered Framework for Few-Shot Knowledge Base Question Answering

arXiv: https://arxiv.org/abs/2308.12060 · HTML: https://mlsanity.com/paper/2308.12060/

## Key facts

- Cited by 51 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 70 references that are in the map.
- First posted on arXiv 23 Aug 2023.

## Builds on (most cited first, 30 of 32)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Modeling Relational Data with Graph Convolutional Networks](https://mlsanity.com/paper/1703.06103/) (2017)
- [ProgPrompt: Generating Situated Robot Task Plans using Large Language Models](https://mlsanity.com/paper/2209.11302/) (2022)
- [LEVER: Learning to Verify Language-to-Code Generation with Execution](https://mlsanity.com/paper/2302.08468/) (2023)
- [Binding Language Models in Symbolic Languages](https://mlsanity.com/paper/2210.02875/) (2022)
- [ZeroGen: Efficient Zero-shot Learning via Dataset Generation](https://mlsanity.com/paper/2202.07922/) (2022)
- [Generating Training Data with Language Models: Towards Zero-Shot Language Understanding](https://mlsanity.com/paper/2202.04538/) (2022)
- [Generating Datasets with Pretrained Language Models](https://mlsanity.com/paper/2104.07540/) (2021)
- [Beyond I.I.D.: Three Levels of Generalization for Question Answering on Knowledge Bases](https://mlsanity.com/paper/2011.07743/) (2020)
- [Case-based Reasoning for Natural Language Queries over Knowledge Bases](https://mlsanity.com/paper/2104.08762/) (2021)
- [FewRel 2.0: Towards More Challenging Few-Shot Relation Classification](https://mlsanity.com/paper/1910.07124/) (2019)
- [RnG-KBQA: Generation Augmented Iterative Ranking for Knowledge Base Question Answering](https://mlsanity.com/paper/2109.08678/) (2021)
- [Batch Prompting: Efficient Inference with Large Language Model APIs](https://mlsanity.com/paper/2301.08721/) (2023)
- [DecAF: Joint Decoding of Answers and Logical Forms for Question Answering over Knowledge Bases](https://mlsanity.com/paper/2210.00063/) (2022)
- [KQA Pro: A Dataset with Explicit Compositional Programs for Complex Question Answering over Knowledge Base](https://mlsanity.com/paper/2007.03875/) (2020)
- [A Survey on Complex Knowledge Base Question Answering: Methods, Challenges and Solutions](https://mlsanity.com/paper/2105.11644/) (2021)
- [Efficient One-Pass End-to-End Entity Linking for Questions](https://mlsanity.com/paper/2010.02413/) (2020)
- [Few-shot In-context Learning for Knowledge Base Question Answering](https://mlsanity.com/paper/2305.01750/) (2023)
- [TIARA: Multi-grained Retrieval for Robust Question Answering over Large Knowledge Bases](https://mlsanity.com/paper/2210.12925/) (2022)
- [Don't Generate, Discriminate: A Proposal for Grounding Language Models to Real-World Environments](https://mlsanity.com/paper/2212.09736/) (2022)
- [ArcaneQA: Dynamic Program Induction and Contextualized Encoding for Knowledge Base Question Answering](https://mlsanity.com/paper/2204.08109/) (2022)
- [Program Transfer for Answering Complex Questions over Knowledge Bases](https://mlsanity.com/paper/2110.05743/) (2021)
- [LINGUIST: Language Model Instruction Tuning to Generate Annotated Utterances for Intent Classification and Slot Tagging](https://mlsanity.com/paper/2209.09900/) (2022)
- [Knowledge Base Question Answering: A Semantic Parsing Perspective](https://mlsanity.com/paper/2209.04994/) (2022)
- [CLASP: Few-Shot Cross-Lingual Data Augmentation for Semantic Parsing](https://mlsanity.com/paper/2210.07074/) (2022)
- [Generating Data for Symbolic Language with Large Language Models](https://mlsanity.com/paper/2305.13917/) (2023)

## Built on by (most cited first, 30 of 51)

- [Next-Generation Database Interfaces: A Survey of LLM-based Text-to-SQL](https://mlsanity.com/paper/2406.08426/) (2024)
- [Plan-on-Graph: Self-Correcting Adaptive Planning of Large Language Model on Knowledge Graphs](https://mlsanity.com/paper/2410.23875/) (2024)
- [KAG: Boosting LLMs in Professional Domains via Knowledge Augmented Generation](https://mlsanity.com/paper/2409.13731/) (2024)
- [UniBind: LLM-Augmented Unified and Balanced Representation Space to Bind Them All](https://mlsanity.com/paper/2403.12532/) (2024)
- [Efficacy of Synthetic Data as a Benchmark](https://mlsanity.com/paper/2409.11968/) (2024)
- [Logically Consistent Language Models via Neuro-Symbolic Integration](https://mlsanity.com/paper/2409.13724/) (2024)
- [Interactive-KBQA: Multi-Turn Interactions for Knowledge Base Question Answering with Large Language Models](https://mlsanity.com/paper/2402.15131/) (2024)
- [A Survey on Large Language Models for Communication, Network, and Service Management: Application Insights, Challenges, and Future Directions](https://mlsanity.com/paper/2412.19823/) (2024)
- [Jailbreak-AudioBench: In-Depth Evaluation and Analysis of Jailbreak Threats for Large Audio Language Models](https://mlsanity.com/paper/2501.13772/) (2025)
- [A Survey of Large Language Models on Generative Graph Analytics: Query, Learning, and Applications](https://mlsanity.com/paper/2404.14809/) (2024)
- [keqing: knowledge-based question answering is a nature chain-of-thought mentor of LLM](https://mlsanity.com/paper/2401.00426/) (2023)
- [Enhancing Complex Question Answering over Knowledge Graphs through Evidence Pattern Retrieval](https://mlsanity.com/paper/2402.02175/) (2024)
- [LLM/Agent-as-Data-Analyst: A Survey](https://mlsanity.com/paper/2509.23988/) (2025)
- [CITING: Large Language Models Create Curriculum for Instruction Tuning](https://mlsanity.com/paper/2310.02527/) (2023)
- [Aligning Large Language Models to a Domain-specific Graph Database for NL2GQL](https://mlsanity.com/paper/2402.16567/) (2024)
- [Assessing SPARQL capabilities of Large Language Models](https://mlsanity.com/paper/2409.05925/) (2024)
- [Leveraging small language models for Text2SPARQL tasks to improve the resilience of AI assistance](https://mlsanity.com/paper/2405.17076/) (2024)
- [OneEdit: A Neural-Symbolic Collaboratively Knowledge Editing System](https://mlsanity.com/paper/2409.07497/) (2024)
- [Bring Your Own KG: Self-Supervised Program Synthesis for Zero-Shot KGQA](https://mlsanity.com/paper/2311.07850/) (2023)
- [Adaptive Pre-training Data Detection for Large Language Models via Surprising Tokens](https://mlsanity.com/paper/2407.21248/) (2024)
- [AdaRefiner: Refining Decisions of Language Models with Adaptive Feedback](https://mlsanity.com/paper/2309.17176/) (2023)
- [FusionFactory: Fusing LLM Capabilities with Multi-LLM Log Data](https://mlsanity.com/paper/2507.10540/) (2025)
- [iQUEST: An Iterative Question-Guided Framework for Knowledge Base Question Answering](https://mlsanity.com/paper/2506.01784/) (2025)
- [Step-Audio-AQAA: a Fully End-to-End Expressive Large Audio Language Model](https://mlsanity.com/paper/2506.08967/) (2025)
- [From Natural Language to Extensive-Form Game Representations](https://mlsanity.com/paper/2501.17282/) (2025)
- [SPARKLE: Enhancing SPARQL Generation with Direct KG Integration in Decoding](https://mlsanity.com/paper/2407.01626/) (2024)
- [TARGA: Targeted Synthetic Data Generation for Practical Reasoning over Structured Data](https://mlsanity.com/paper/2412.19544/) (2024)
- [Large Language Model in Financial Regulatory Interpretation](https://mlsanity.com/paper/2405.06808/) (2024)
- [From Anchors to Answers: A Novel Node Tokenizer for Integrating Graph Structure into Large Language Models](https://mlsanity.com/paper/2410.10743/) (2024)
- [Knowledge-Level Consistency Reinforcement Learning: Dual-Fact Alignment for Long-Form Factuality](https://mlsanity.com/paper/2509.23765/) (2025)

## Abstract (arXiv)

Knowledge base question answering (KBQA) is a critical yet challenging task due to the vast number of entities within knowledge bases and the diversity of natural language questions posed by users. Unfortunately, the performance of most KBQA models tends to decline significantly in real-world scenarios where high-quality annotated data is insufficient. To mitigate the burden associated with manual annotation, we introduce FlexKBQA by utilizing Large Language Models (LLMs) as program translators for addressing the challenges inherent in the few-shot KBQA task. Specifically, FlexKBQA leverages automated algorithms to sample diverse programs, such as SPARQL queries, from the knowledge base, which are subsequently converted into natural language questions via LLMs. This synthetic dataset facilitates training a specialized lightweight model for the KB. Additionally, to reduce the barriers of distribution shift between synthetic data and real user questions, FlexKBQA introduces an executionguided self-training method to iterative leverage unlabeled user questions. Furthermore, we explore harnessing the inherent reasoning capability of LLMs to enhance the entire framework. Consequently, FlexKBQA delivers substantial flexibility, encompassing data annotation, deployment, and being domain agnostic. Through extensive experiments on GrailQA, WebQSP, and KQA Pro, we observe that under the few-shot even the more challenging zero-shot scenarios, FlexKBQA achieves impressive results with a few annotations, surpassing all previous baselines and even approaching the performance of supervised models, achieving a remarkable 93% performance relative to the fully-supervised models. We posit that FlexKBQA represents a significant advancement towards exploring better integration of large and lightweight models. The code is open-sourced.