mlsanityData to 7 Oct 2026

Knowledge-Augmented Language Model Prompting for Zero-Shot Knowledge Graph Question Answering

arXiv 2306.04136 ↗First posted 7 Jun 2023cs.CLcited by 119

Not on the map. Why?

Cited by 34 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large Language Models (LLMs) are capable of performing zero-shot closed-book question answering tasks, based on their internal knowledge stored in parameters during pre-training. However, such internalized knowledge might be insufficient and incorrect, which could lead LLMs to generate factually wrong answers. Furthermore, fine-tuning LLMs to update their knowledge is expensive. To this end, we propose to augment the knowledge directly in the input of LLMs. Specifically, we first retrieve the relevant facts to the input question from the knowledge graph based on semantic similarities between the question and its associated facts. After that, we prepend the retrieved facts to the input question in the form of the prompt, which is then forwarded to LLMs to generate the answer. Our framework, Knowledge-Augmented language model PromptING (KAPING), requires no model training, thus completely zero-shot. We validate the performance of our KAPING framework on the knowledge graph question answering task, that aims to answer the user's question based on facts over a knowledge graph, on which ours outperforms relevant zero-shot baselines by up to 48% in average, across multiple LLMs of various sizes.

Key facts. Cited by 119 papers in the map (data through 7 Oct 2026). Builds on 57 of its 80 references that are in the map. First posted on arXiv 7 Jun 2023.

Builds on

57 of 80 references.

References · 57

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  8. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  9. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  10. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  11. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  12. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
38 more
  1. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  2. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  3. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  4. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  5. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  6. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  7. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  8. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  9. 2019Language Models as Knowledge Bases?cited by 1,922
  10. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  11. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  12. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  13. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  14. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  15. 2016ConceptNet 5.5: An Open Multilingual Graph of General Knowledgecited by 1,065
  16. 2020MPNet: Masked and Permuted Pre-training for Language Understandingcited by 973
  17. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  18. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 846
  19. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 713
  20. 2018Object Hallucination in Image Captioningcited by 661
  21. 2021Learning To Retrieve Prompts for In-Context Learningcited by 641
  22. 2021QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answeringcited by 386
  23. 2020Pretrained Transformers for Text Ranking: BERT and Beyondcited by 266
  24. 2021Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Samplingcited by 238
  25. 2021Generated Knowledge Prompting for Commonsense Reasoningcited by 213
  26. 2021Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classificationcited by 183
  27. 2021KnowPrompt: Knowledge-aware Prompt-tuning with Synergistic Optimization for Relation Extractioncited by 180
  28. 2018Open Domain Question Answering Using Early Fusion of Knowledge Bases and Textcited by 167
  29. 2019Scalable Zero-shot Entity Linking with Dense Entity Retrievalcited by 85
  30. 2021Can Language Models be Biomedical Knowledge Bases?cited by 83
  31. 2020Efficient One-Pass End-to-End Entity Linking for Questionscited by 76
  32. 2022ReFinED: An Efficient Zero-shot-capable Approach to End-to-End Entity Linkingcited by 74
  33. 2022Mintaka: A Complex, Natural, and Multilingual Dataset for End-to-End Question Answeringcited by 71
  34. 2022Multi-hop Question Answeringcited by 64
  35. 2020UniK-QA: Unified Representations of Structured and Unstructured Knowledge for Open-Domain Question Answeringcited by 61
  36. 2022AlexaTM 20B: Few-Shot Learning Using a Large-Scale Multilingual Seq2Seq Modelcited by 57
  37. 2020A Survey on Complex Question Answering over Knowledge Base: Recent Advances and Challengescited by 47
  38. 2022Decoupling Knowledge from Memorization: Retrieval-augmented Prompt Learningcited by 36

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 13; 2024: 56; 2025: 35; 2026: 15.

119 papers.

Sort:

By area:

  1. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  2. 2023Think-on-Graph: Deep and Responsible Reasoning of Large Language Model on Knowledge Graphcited by 235
  3. 2024ResearchAgent: Iterative Research Idea Generation over Scientific Literature with Large Language Modelscited by 200
  4. 2024Retrieval-Augmented Generation with Graphs (GraphRAG)cited by 170
  5. 2024AriGraph: Learning Knowledge Graph World Models with Episodic Memory for LLM Agentscited by 100
  6. 2023Can Knowledge Graphs Reduce Hallucinations in LLMs? : A Surveycited by 92
  7. 2024GRAG: Graph Retrieval-Augmented Generationcited by 77
  8. 2023MindMap: Knowledge Graph Prompting Sparks Graph of Thoughts in Large Language Modelscited by 73
  9. 2024Retrieval-Augmented Generation for Natural Language Processing: A Surveycited by 70
  10. 2025Knowledge Graph Retrieval-Augmented Generation for LLM-based Recommendationcited by 56
  11. 2023KnowledGPT: Enhancing Large Language Models with Retrieval and Storage Access on Knowledge Basescited by 52
  12. 2023Making Large Language Models Perform Better in Knowledge Graph Completioncited by 48
38 more
  1. 2024Paths-over-Graph: Knowledge Graph Empowered Large Language Model Reasoningcited by 44
  2. 2023Knowledge-Augmented Large Language Models for Personalized Contextual Query Suggestioncited by 43
  3. 2024Multimodal Reasoning with Multimodal Knowledge Graphcited by 43
  4. 2024FiDeLiS: Faithful Reasoning in Large Language Model for Knowledge Graph Question Answeringcited by 38
  5. 2024Graph Machine Learning in the Era of Large Language Models (LLMs)cited by 33
  6. 2023GenTKG: Generative Forecasting on Temporal Knowledge Graph with Large Language Modelscited by 30
  7. 2023Trends in Integration of Knowledge and Large Language Models: A Survey and Taxonomy of Methods, Benchmarks, and Applicationscited by 28
  8. 2025NodeRAG: Structuring Graph-based RAG with Heterogeneous Nodescited by 25
  9. 2023KnowledgeNavigator: Leveraging Large Language Models for Enhanced Reasoning over Knowledge Graphcited by 22
  10. 2024Can Knowledge Graphs Make Large Language Models More Trustworthy? An Empirical Study Over Open-ended Question Answeringcited by 20
  11. 2024Triad: A Framework Leveraging a Multi-Role LLM-based Agent to Solve Knowledge Base Question Answeringcited by 18
  12. 2024Two-stage Generative Question Answering on Temporal Knowledge Graph Using Large Language Modelscited by 18
  13. 2024LLM-Based Multi-Hop Question Answering with Knowledge Graph Integration in Evolving Environmentscited by 18
  14. 2024GraphInsight: Unlocking Insights in Large Language Models for Graph Structure Understandingcited by 18
  15. 2025Graph-based Approaches and Functionalities in Retrieval-Augmented Generation: A Comprehensive Surveycited by 18
  16. 2024Combining Knowledge Graphs and Large Language Modelscited by 16
  17. 2024Cross-Data Knowledge Graph Construction for LLM-enabled Educational Question-Answering System: A Case Study at HCMUTcited by 14
  18. 2024EffiQA: Efficient Question-Answering with Strategic Multi-Model Collaboration on Knowledge Graphscited by 14
  19. 2024Research Trends for the Interplay between Large Language Models and Knowledge Graphscited by 14
  20. 2025Knowledge Graph-extended Retrieval Augmented Generation for Question Answeringcited by 14
  21. 2023Code-Style In-Context Learning for Knowledge-Based Question Answeringcited by 13
  22. 2024EWEK-QA: Enhanced Web and Efficient Knowledge Graph Retrieval for Citation-based Question Answering Systemscited by 13
  23. 2024Question-guided Knowledge Graph Re-scoring and Injection for Knowledge Graph Question Answeringcited by 13
  24. 2025VistaWise: Building Cost-Effective Agent with Cross-Modal Knowledge Graph for Minecraftcited by 13
  25. 2024Mindful-RAG: A Study of Points of Failure in Retrieval Augmented Generationcited by 12
  26. 2024Evaluating Deep Unlearning in Large Language Modelscited by 12
  27. 2025GraphCheck: Breaking Long-Term Text Barriers with Extracted Knowledge Graph-Powered Fact-Checkingcited by 12
  28. 2023Bring Your Own KG: Self-Supervised Program Synthesis for Zero-Shot KGQAcited by 11
  29. 2024Right for Right Reasons: Large Language Models for Verifiable Commonsense Knowledge Graph Question Answeringcited by 11
  30. 2024Retrieval-Enhanced Machine Learning: Synthesis and Opportunitiescited by 11
  31. 2024CommunityKG-RAG: Leveraging Community Structures in Knowledge Graphs for Advanced Retrieval-Augmented Generation in Fact-Checkingcited by 11
  32. 2025LLM-Friendly Knowledge Representation for Customer Supportcited by 11
  33. 2024Large Language Models Can Better Understand Knowledge Graphs Than We Thoughtcited by 10
  34. 2024KnowLA: Enhancing Parameter-efficient Finetuning with Knowledgeable Adaptationcited by 10
  35. 2024Graph Neural Network Enhanced Retrieval for Question Answering of LLMscited by 10
  36. 2025Out of Style: RAG's Fragility to Linguistic Variationcited by 10
  37. 2024Evidence-Focused Fact Summarization for Knowledge-Augmented Zero-Shot Question Answeringcited by 9
  38. 2024On-the-fly Definition Augmentation of LLMs for Biomedical NERcited by 9

50 most cited of 119 All 119 (JSON)