mlsanityData to 7 Oct 2026

keqing: knowledge-based question answering is a nature chain-of-thought mentor of LLM

arXiv 2401.00426 ↗First posted 31 Dec 2023cs.CL cs.AIcited by 16

On the map: LLM reasoning → Knowledge- and tool-augmented reasoning

Abstract (arXiv)

Large language models (LLMs) have exhibited remarkable performance on various natural language processing (NLP) tasks, especially for question answering. However, in the face of problems beyond the scope of knowledge, these LLMs tend to talk nonsense with a straight face, where the potential solution could be incorporating an Information Retrieval (IR) module and generating response based on these retrieved knowledge. In this paper, we present a novel framework to assist LLMs, such as ChatGPT, to retrieve question-related structured information on the knowledge graph, and demonstrate that Knowledge-based question answering (Keqing) could be a nature Chain-of-Thought (CoT) mentor to guide the LLM to sequentially find the answer entities of a complex question through interpretable logical chains. Specifically, the workflow of Keqing will execute decomposing a complex question according to predefined templates, retrieving candidate entities on knowledge graph, reasoning answers of sub-questions, and finally generating response with reasoning paths, which greatly improves the reliability of LLM's response. The experimental results on KBQA datasets show that Keqing can achieve competitive performance and illustrate the logic of answering each question.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning. Builds on 37 of its 43 references that are in the map. First posted on arXiv 31 Dec 2023.

Builds on

37 of 43 references.

References · 37

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  8. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  10. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  11. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  12. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
25 more
  1. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  2. 2023A Survey of Large Language Modelscited by 2,537
  3. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  4. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  5. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  6. 2023In-Context Retrieval-Augmented Language Modelscited by 694
  7. 2023Augmented Language Models: a Surveycited by 398
  8. 2023StructGPT: A General Framework for Large Language Model to Reason over Structured Datacited by 339
  9. 2022Binding Language Models in Symbolic Languagescited by 224
  10. 2016Key-Value Memory Networks for Directly Reading Documentscited by 223
  11. 2019PullNet: Open Domain Question Answering with Iterative Retrieval on Knowledge Bases and Textcited by 182
  12. 2017Variational Reasoning for Question Answering with Knowledge Graphcited by 170
  13. 2023Sources of Hallucination by Large Language Models on Inference Taskscited by 168
  14. 2018Open Domain Question Answering Using Early Fusion of Knowledge Bases and Textcited by 167
  15. 2021Exploring the Limits of Large Scale Pre-trainingcited by 104
  16. 2021Improving Multi-hop Knowledge Base Question Answering by Learning Intermediate Supervision Signalscited by 100
  17. 2022DecAF: Joint Decoding of Answers and Logical Forms for Question Answering over Knowledge Basescited by 83
  18. 2023Few-shot In-context Learning for Knowledge Base Question Answeringcited by 74
  19. 2020UniK-QA: Unified Representations of Structured and Unstructured Knowledge for Open-Domain Question Answeringcited by 61
  20. 2022TIARA: Multi-grained Retrieval for Robust Question Answering over Large Knowledge Basescited by 59
  21. 2022Don't Generate, Discriminate: A Proposal for Grounding Language Models to Real-World Environmentscited by 57
  22. 2023FlexKBQA: A Flexible LLM-Powered Framework for Few-Shot Knowledge Base Question Answeringcited by 51
  23. 2021One Question Answering Model for Many Languages with Cross-lingual Dense Passage Retrievalcited by 46
  24. 2022Knowledge Base Question Answering by Case-based Reasoning over Subgraphscited by 31
  25. 2022Question-Answer Sentence Graph for Joint Modeling Answer Selectioncited by 6

Built on by

Cited by year

Citing papers per year: 2024: 12; 2025: 4.

16 papers.

Sort:

By area:

  1. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 352
  2. 2024Graph Retrieval-Augmented Generation: A Surveycited by 320
  3. 2024Retrieval-Augmented Generation with Graphs (GraphRAG)cited by 170
  4. 2024Generate-on-Graph: Treat LLM as both Agent and KG in Incomplete Knowledge Graph Question Answeringcited by 59
  5. 2024Crafting Personalized Agents through Retrieval-Augmented Generation on Editable Memory Graphscited by 38
  6. 2024Graph Machine Learning in the Era of Large Language Models (LLMs)cited by 33
  7. 2025Large Language Models Meet Knowledge Graphs for Question Answering: Synthesis and Opportunitiescited by 30
  8. 2025Graph-based Approaches and Functionalities in Retrieval-Augmented Generation: A Comprehensive Surveycited by 18
  9. 2024medIKAL: Integrating Knowledge Graphs as Assistants of LLMs for Enhanced Clinical Diagnosis on EMRscited by 16
  10. 2025Knowledge Graph-extended Retrieval Augmented Generation for Question Answeringcited by 14
  11. 2024ODA: Observation-Driven Agent for integrating LLMs and Knowledge Graphscited by 8
  12. 2024Exploring Prompt Engineering: A Systematic Review with SWOT Analysiscited by 5
4 more
  1. 2025Reasoning by Exploration: A Unified Approach to Retrieval and Generation over Graphscited by 5
  2. 2024Pyramid-Driven Alignment: Pyramid Principle Guided Integration of Large Language Models and Knowledge Graphscited by 2
  3. 2024A Learn-Then-Reason Model Towards Generalization in Knowledge Base Question Answeringcited by 0
  4. 2024Uncovering Autoregressive LLM Knowledge of Thematic Fit in Event Representationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Beyond Chains: Bridging Large Language Models and Knowledge Bases in Complex Question Answering
  2. 2025Fusing Knowledge and Language: A Comparative Study of Knowledge Graph-Based Question Answering with LLMs
  3. 2025RJE: A Retrieval-Judgment-Exploration Framework for Efficient Knowledge Graph Question Answering with LLMs
  4. 2025KERAG: Knowledge-Enhanced Retrieval-Augmented Generation for Advanced Question Answering
  5. 2024Decoding on Graphs: Faithful and Sound Reasoning on Knowledge Graphs through Generation of Well-Formed Chains
  6. 2024FiDeLiS: Faithful Reasoning in Large Language Model for Knowledge Graph Question Answering
  7. 2025The Role of Exploration Modules in Small Language Models for Knowledge Graph Question Answering
  8. 2024Logic Query of Thoughts: Guiding Large Language Models to Answer Complex Logic Queries with Knowledge Graphs