# Evaluating ChatGPT as a Question Answering System: A Comprehensive Analysis and Comparison with Existing Models

arXiv: https://arxiv.org/abs/2312.07592 · HTML: https://mlsanity.com/paper/2312.07592/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 22 references that are in the map.
- First posted on arXiv 11 Dec 2023.

## Builds on (most cited first, 14 of 14)

- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [Know What You Don't Know: Unanswerable Questions for SQuAD](https://mlsanity.com/paper/1806.03822/) (2018)
- [SpanBERT: Improving Pre-training by Representing and Predicting Spans](https://mlsanity.com/paper/1907.10529/) (2019)
- [NewsQA: A Machine Comprehension Dataset](https://mlsanity.com/paper/1611.09830/) (2016)
- [LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention](https://mlsanity.com/paper/2010.01057/) (2020)
- [Retrieving and Reading: A Comprehensive Survey on Open-domain Question Answering](https://mlsanity.com/paper/2101.00774/) (2021)
- [Improving the Domain Adaptation of Retrieval Augmented Generation (RAG) Models for Open Domain Question Answering](https://mlsanity.com/paper/2210.02627/) (2022)
- [Retrospective Reader for Machine Reading Comprehension](https://mlsanity.com/paper/2001.09694/) (2020)
- [ChatGPT versus Traditional Question Answering for Knowledge Graphs: Current Status and Future Directions Towards Knowledge Graph Chatbots](https://mlsanity.com/paper/2302.06466/) (2023)
- [A Question-Focused Multi-Factor Attention Network for Question Answering](https://mlsanity.com/paper/1801.08290/) (2018)
- [Densely Connected Attention Propagation for Reading Comprehension](https://mlsanity.com/paper/1811.04210/) (2018)

## Built on by (most cited first, 11 of 11)

- [MAiDE-up: Multilingual Deception Detection of GPT-generated Hotel Reviews](https://mlsanity.com/paper/2404.12938/) (2024)
- [Exploring the Escalation of Source Bias in User, Data, and Recommender System Feedback Loop](https://mlsanity.com/paper/2405.17998/) (2024)
- [Red Skills or Blue Skills? A Dive Into Skills Published on ClawHub](https://mlsanity.com/paper/2604.13064/) (2026)
- [It Is Not About What You Say, It Is About How You Say It: A Surprisingly Simple Approach for Improving Reading Comprehension](https://mlsanity.com/paper/2406.16779/) (2024)
- [Structure-Augmented Reasoning Generation](https://mlsanity.com/paper/2506.08364/) (2025)
- [LP-LM: No Hallucinations in Question Answering with Logic Programming](https://mlsanity.com/paper/2502.09212/) (2025)
- [DaVinci at SemEval-2024 Task 9: Few-shot prompting GPT-3.5 for Unconventional Reasoning](https://mlsanity.com/paper/2405.11559/) (2024)
- [Is Our Chatbot Telling Lies? Assessing Correctness of an LLM-based Dutch Support Chatbot](https://mlsanity.com/paper/2411.00034/) (2024)
- [PersianRAG: A Retrieval-Augmented Generation System for Persian Language](https://mlsanity.com/paper/2411.02832/) (2024)
- [A Reproducibility Study of Metacognitive Retrieval-Augmented Generation](https://mlsanity.com/paper/2604.19899/) (2026)
- [Assessing Reliability of BERT-Based Models on Question Answering Tasks](https://mlsanity.com/paper/2608.10806/) (2026)

## Abstract (arXiv)

In the current era, a multitude of language models has emerged to cater to user inquiries. Notably, the GPT-3.5 Turbo language model has gained substantial attention as the underlying technology for ChatGPT. Leveraging extensive parameters, this model adeptly responds to a wide range of questions. However, due to its reliance on internal knowledge, the accuracy of responses may not be absolute. This article scrutinizes ChatGPT as a Question Answering System (QAS), comparing its performance to other existing QASs. The primary focus is on evaluating ChatGPT's proficiency in extracting responses from provided paragraphs, a core QAS capability. Additionally, performance comparisons are made in scenarios without a surrounding passage. Multiple experiments, exploring response hallucination and considering question complexity, were conducted on ChatGPT. Evaluation employed well-known Question Answering (QA) datasets, including SQuAD, NewsQA, and PersianQuAD, across English and Persian languages. Metrics such as F-score, exact match, and accuracy were employed in the assessment. The study reveals that, while ChatGPT demonstrates competence as a generative model, it is less effective in question answering compared to task-specific models. Providing context improves its performance, and prompt engineering enhances precision, particularly for questions lacking explicit answers in provided paragraphs. ChatGPT excels at simpler factual questions compared to "how" and "why" question types. The evaluation highlights occurrences of hallucinations, where ChatGPT provides responses to questions without available answers in the provided context.