# ChatGPT-Crawler: Find out if ChatGPT really knows what it's talking about

arXiv: https://arxiv.org/abs/2304.03325 · HTML: https://mlsanity.com/paper/2304.03325/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 35 references that are in the map.
- First posted on arXiv 6 Apr 2023.

## Builds on (most cited first, 17 of 17)

- [A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity](https://mlsanity.com/paper/2302.04023/) (2023)
- [BARTScore: Evaluating Generated Text as Text Generation](https://mlsanity.com/paper/2106.11520/) (2021)
- [CoQA: A Conversational Question Answering Challenge](https://mlsanity.com/paper/1808.07042/) (2018)
- [Is ChatGPT a General-Purpose Natural Language Processing Task Solver?](https://mlsanity.com/paper/2302.06476/) (2023)
- [How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection](https://mlsanity.com/paper/2301.07597/) (2023)
- [Regulating ChatGPT and other Large Generative AI Models](https://mlsanity.com/paper/2302.02337/) (2023)
- [ChatGPT Makes Medicine Easy to Swallow: An Exploratory Case Study on Simplified Radiology Reports](https://mlsanity.com/paper/2212.14882/) (2022)
- [ChatGPT: The End of Online Exam Integrity?](https://mlsanity.com/paper/2212.09292/) (2022)
- [ChatGPT is not all you need. A State of the Art Review of large Generative AI models](https://mlsanity.com/paper/2301.04655/) (2023)
- [Can ChatGPT Assess Human Personalities? A General Evaluation Framework](https://mlsanity.com/paper/2303.01248/) (2023)
- [DialFact: A Benchmark for Fact-Checking in Dialogue](https://mlsanity.com/paper/2110.08222/) (2021)
- [Will ChatGPT get you caught? Rethinking of Plagiarism Detection](https://mlsanity.com/paper/2302.04335/) (2023)
- [FaVIQ: FAct Verification from Information-seeking Questions](https://mlsanity.com/paper/2107.02153/) (2021)
- ["I think this is the most disruptive technology": Exploring Sentiments of ChatGPT Early Adopters using Twitter Data](https://mlsanity.com/paper/2212.05856/) (2022)
- [Towards Human-Bot Collaborative Software Architecting with ChatGPT](https://mlsanity.com/paper/2302.14600/) (2023)
- [Jaccard Metric Losses: Optimizing the Jaccard Index with Soft Labels](https://mlsanity.com/paper/2302.05666/) (2023)
- [UZH_CLyp at SemEval-2023 Task 9: Head-First Fine-Tuning and ChatGPT Data Generation for Cross-Lingual Learning in Tweet Intimacy Prediction](https://mlsanity.com/paper/2303.01194/) (2023)

## Built on by (most cited first, 3 of 3)

- [Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMs](https://mlsanity.com/paper/2402.03927/) (2024)
- [A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasets](https://mlsanity.com/paper/2305.18486/) (2023)
- [Knowing What LLMs DO NOT Know: A Simple Yet Effective Self-Detection Method](https://mlsanity.com/paper/2310.17918/) (2023)

## Abstract (arXiv)

Large language models have gained considerable interest for their impressive performance on various tasks. Among these models, ChatGPT developed by OpenAI has become extremely popular among early adopters who even regard it as a disruptive technology in many fields like customer service, education, healthcare, and finance. It is essential to comprehend the opinions of these initial users as it can provide valuable insights into the potential strengths, weaknesses, and success or failure of the technology in different areas. This research examines the responses generated by ChatGPT from different Conversational QA corpora. The study employed BERT similarity scores to compare these responses with correct answers and obtain Natural Language Inference(NLI) labels. Evaluation scores were also computed and compared to determine the overall performance of GPT-3 & GPT-4. Additionally, the study identified instances where ChatGPT provided incorrect answers to questions, providing insights into areas where the model may be prone to error.