mlsanityData to 7 Oct 2026

Can ChatGPT Replace Traditional KBQA Models? An In-depth Analysis of the Question Answering Performance of the GPT LLM Family

arXiv 2303.07992 ↗First posted 14 Mar 2023cs.CLcited by 71

Not on the map. Why?

Abstract (arXiv)

ChatGPT is a powerful large language model (LLM) that covers knowledge resources such as Wikipedia and supports natural language question answering using its own knowledge. Therefore, there is growing interest in exploring whether ChatGPT can replace traditional knowledge-based question answering (KBQA) models. Although there have been some works analyzing the question answering performance of ChatGPT, there is still a lack of large-scale, comprehensive testing of various types of complex questions to analyze the limitations of the model. In this paper, we present a framework that follows the black-box testing specifications of CheckList proposed by Ribeiro et. al. We evaluate ChatGPT and its family of LLMs on eight real-world KB-based complex question answering datasets, which include six English datasets and two multilingual datasets. The total number of test cases is approximately 190,000. In addition to the GPT family of LLMs, we also evaluate the well-known FLAN-T5 to identify commonalities between the GPT family and other LLMs. The dataset and code are available at https://github.com/tan92hl/Complex-Question-Answering-Evaluation-of-GPT-family.git

Key facts. Cited by 71 papers in the map (data through 7 Oct 2026). Builds on 35 of its 55 references that are in the map. First posted on arXiv 14 Mar 2023.

Builds on

35 of 55 references.

References · 35

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  8. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  9. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  10. 2019Language Models as Knowledge Bases?cited by 1,922
  11. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  12. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
23 more
  1. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  2. 2019How Can We Know What Language Models Know?cited by 953
  3. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 888
  4. 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 599
  5. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 486
  6. 2018The Web as a Knowledge-base for Answering Complex Questionscited by 409
  7. 2021Learning How to Ask: Querying LMs with Mixtures of Soft Promptscited by 376
  8. 2018Analysis Methods in Neural Language Processing: A Surveycited by 276
  9. 2023ChatGPT: Jack of all trades, master of nonecited by 204
  10. 2020Beyond I.I.D.: Three Levels of Generalization for Question Answering on Knowledge Basescited by 169
  11. 2023Is ChatGPT better than Human Annotators? Potential and Limitations of ChatGPT in Explaining Implicit Hate Speechcited by 165
  12. 2023Can ChatGPT Understand Too? A Comparative Study on ChatGPT and Fine-tuned BERTcited by 155
  13. 2020MKQA: A Linguistically Diverse Benchmark for Multilingual Open Domain Question Answeringcited by 151
  14. 2021RnG-KBQA: Generation Augmented Iterative Ranking for Knowledge Base Question Answeringcited by 86
  15. 2020KQA Pro: A Dataset with Explicit Compositional Programs for Complex Question Answering over Knowledge Basecited by 76
  16. 2023How Robust is GPT-3.5 to Predecessors? A Comprehensive Study on Language Understanding Taskscited by 58
  17. 2023Can ChatGPT Write a Good Boolean Query for Systematic Review Literature Search?cited by 54
  18. 2022ArcaneQA: Dynamic Program Induction and Contextualized Encoding for Knowledge Base Question Answeringcited by 48
  19. 2023ChatGPT versus Traditional Question Answering for Knowledge Graphs: Current Status and Future Directions Towards Knowledge Graph Chatbotscited by 47
  20. 2021The Stem Cell Hypothesis: Dilemma behind Multi-Task Learning with Transformer Encoderscited by 31
  21. 2021UNIQORN: Unified Question Answering over RDF Knowledge Graphs and Natural Language Textcited by 17
  22. 2022Knowledge Graph Question Answering Leaderboard: A Community Resource to Prevent a Replication Crisiscited by 14
  23. 2022GraphQ IR: Unifying the Semantic Parsing of Graph Query Languages with One Intermediate Representationcited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 7; 2024: 39; 2025: 19; 2026: 6.

71 papers.

Sort:

By area:

  1. 2024Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMscited by 224
  2. 2023Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificitycited by 204
  3. 2023Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMscited by 175
  4. 2024Multi-Step Reasoning with Large Language Models, a Surveycited by 140
  5. 2023LLMs for Knowledge Graph Construction and Reasoning: Recent Capabilities and Future Opportunitiescited by 97
  6. 2024Reasoning Aware Self-Consistency: Leveraging Reasoning Paths for Efficient LLM Samplingcited by 63
  7. 2024Don't Forget to Connect! Improving RAG with Graph-based Rerankingcited by 49
  8. 2024Usable XAI: 10 Strategies Towards Exploiting Explainability in the LLM Eracited by 45
  9. 2024CoRAL: Collaborative Retrieval-Augmented Large Language Models Improve Long-tail Recommendationcited by 44
  10. 2024Call Me When Necessary: LLMs can Efficiently and Faithfully Reason over Structured Environmentscited by 27
  11. 2024SPINACH: SPARQL-Based Information Navigation for Challenging Real-World Questionscited by 27
  12. 2024Triad: A Framework Leveraging a Multi-Role LLM-based Agent to Solve Knowledge Base Question Answeringcited by 18
38 more
  1. 2024Text Generation: A Systematic Literature Review of Tasks, Evaluation, and Challengescited by 18
  2. 2023Why Can Large Language Models Generate Correct Chain-of-Thoughts?cited by 16
  3. 2024PAT-Questions: A Self-Updating Benchmark for Present-Anchored Temporal Question-Answeringcited by 15
  4. 2024See What LLMs Cannot Answer: A Self-Challenge Framework for Uncovering LLM Weaknessescited by 15
  5. 2024Better to Ask in English: Evaluation of Large Language Models on English, Low-resource and Cross-Lingual Settingscited by 14
  6. 2024Enhancing textual textbook question answering with large language models and retrieval augmented generationcited by 13
  7. 2024EWEK-QA: Enhanced Web and Efficient Knowledge Graph Retrieval for Citation-based Question Answering Systemscited by 13
  8. 2024Mindful-RAG: A Study of Points of Failure in Retrieval Augmented Generationcited by 12
  9. 2024Understanding Multimodal LLMs: the Mechanistic Interpretability of Llava in Visual Question Answeringcited by 11
  10. 2024Large Language Models Can Better Understand Knowledge Graphs Than We Thoughtcited by 10
  11. 2024Evaluating the Factuality of Large Language Models using Large-Scale Knowledge Graphscited by 10
  12. 2025The Bias is in the Details: An Assessment of Cognitive Bias in LLMscited by 10
  13. 2025QuCo-RAG: Quantifying Uncertainty from the Pre-training Corpus for Dynamic Retrieval-Augmented Generationcited by 10
  14. 2025Inverse IFEval: Can LLMs Unlearn Stubborn Training Conventions to Follow Real Instructions?cited by 9
  15. 2023Make a Choice! Knowledge Base Question Answering with In-Context Learningcited by 8
  16. 2025Scaling Down Text Encoders of Text-to-Image Diffusion Modelscited by 8
  17. 2024Multi-hop Question Answering over Knowledge Graphs using Large Language Modelscited by 6
  18. 2024Do Large Language Models Speak All Languages Equally? A Comparative Study in Low-Resource Settingscited by 6
  19. 2024AHP-Powered LLM Reasoning for Multi-Criteria Evaluation of Open-Ended Responsescited by 6
  20. 2024What do Large Language Models Need for Machine Translation Evaluation?cited by 6
  21. 2025From Natural Language to Extensive-Form Game Representationscited by 6
  22. 2023Enhancing In-Context Learning with Answer Feedback for Multi-Span Question Answeringcited by 5
  23. 2024Hire Me or Not? Examining Language Model's Behavior with Occupation Attributescited by 5
  24. 2024Exploring Prompt Engineering: A Systematic Review with SWOT Analysiscited by 5
  25. 2024DRS: Deep Question Reformulation With Structured Outputcited by 5
  26. 2024Self-adaptive Multimodal Retrieval-Augmented Generationcited by 4
  27. 2025CodeEdu: A Multi-Agent Collaborative Platform for Personalized Coding Educationcited by 4
  28. 2025Semantic Reformulation Entropy for Robust Hallucination Detection in QA Taskscited by 4
  29. 2025StyleBench: Evaluating thinking styles in Large Language Modelscited by 4
  30. 2024Leveraging Large Language Models for Concept Graph Recovery and Question Answering in NLP Educationcited by 3
  31. 2024LLM+KG@VLDB'24 Workshop Summarycited by 3
  32. 2025Ontology-Guided, Hybrid Prompt Learning for Generalization in Knowledge Graph Question Answeringcited by 3
  33. 2025DeepSeek: Paradigm Shifts and Technical Evolution in Large AI Modelscited by 3
  34. 2024ProSwitch: Knowledge-Guided Instruction Tuning to Switch Between Professional and Non-Professional Responsescited by 2
  35. 2024Matmul or No Matmul in the Era of 1-bit LLMscited by 2
  36. 2025Information Retrieval in the Age of Generative AI: The RGB Modelcited by 2
  37. 2025HaluNet: Learning Hallucination Risk from Internal Signals in LLM Question Answeringcited by 2
  38. 2026Multi-hop Reasoning and Retrieval in Embedding Space: Leveraging Large Language Models with Knowledgecited by 2

50 most cited of 71 All 71 (JSON)