mlsanityData to 7 Oct 2026

How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection

arXiv 2301.07597 ↗First posted 18 Jan 2023cs.CLcited by 472

Not on the map. Why?

Abstract (arXiv)

The introduction of ChatGPT has garnered widespread attention in both academic and industrial communities. ChatGPT is able to respond effectively to a wide range of human questions, providing fluent and comprehensive answers that significantly surpass previous public chatbots in terms of security and usefulness. On one hand, people are curious about how ChatGPT is able to achieve such strength and how far it is from human experts. On the other hand, people are starting to worry about the potential negative impacts that large language models (LLMs) like ChatGPT could have on society, such as fake news, plagiarism, and social security issues. In this work, we collected tens of thousands of comparison responses from both human experts and ChatGPT, with questions ranging from open-domain, financial, medical, legal, and psychological areas. We call the collected dataset the Human ChatGPT Comparison Corpus (HC3). Based on the HC3 dataset, we study the characteristics of ChatGPT's responses, the differences and gaps from human experts, and future directions for LLMs. We conducted comprehensive human evaluations and linguistic analyses of ChatGPT-generated content compared with that of humans, where many interesting results are revealed. After that, we conduct extensive experiments on how to effectively detect whether a certain text is generated by ChatGPT or humans. We build three different detection systems, explore several key factors that influence their effectiveness, and evaluate them in different scenarios. The dataset, code, and models are all publicly available at https://github.com/Hello-SimpleAI/chatgpt-comparison-detection.

Key facts. Cited by 472 papers in the map (data through 7 Oct 2026). Builds on 24 of its 49 references that are in the map. First posted on arXiv 18 Jan 2023.

Builds on

24 of 49 references.

References · 24

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2017Deep reinforcement learning from human preferencescited by 3,846
  7. 2020Learning to summarize from human feedbackcited by 2,562
  8. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  9. 2019ELI5: Long Form Question Answeringcited by 567
  10. 2019Release Strategies and the Social Impacts of Language Modelscited by 505
  11. 2019GLTR: Statistical Detection and Visualization of Generated Textcited by 490
  12. 2018Plan-And-Write: Towards Better Automatic Storytellingcited by 232
12 more
  1. 2022ADBench: Anomaly Detection Benchmarkcited by 230
  2. 2020Automatic Detection of Machine Generated Text: A Critical Surveycited by 160
  3. 2020MedDialog: Two Large-scale Medical Dialogue Datasetscited by 157
  4. 2020TweepFake: about Detecting Deepfake Tweetscited by 117
  5. 2021XLM-T: Multilingual Language Models in Twitter for Sentiment Analysis and Beyondcited by 114
  6. 2022ChatGPT Makes Medicine Easy to Swallow: An Exploratory Case Study on Simplified Radiology Reportscited by 100
  7. 2022ChatGPT: The End of Online Exam Integrity?cited by 88
  8. 2022Fengshenbang 1.0: Being the Foundation of Chinese Cognitive Intelligencecited by 70
  9. 2022A fine-grained comparison of pragmatic language understanding in humans and language modelscited by 69
  10. 2022Deepfake Text Detection: Limitations and Opportunitiescited by 37
  11. 2022GENIUS: Sketch-based Language Model Pre-training via Extreme and Selective Masking for Text Generation and Augmentationcited by 11
  12. 2022Selective Text Augmentation with Word Roles for Low-Resource Text Classificationcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 159; 2024: 133; 2025: 101; 2026: 79.

472 papers.

Sort:

By area:

  1. 2023A Survey of Large Language Modelscited by 2,537
  2. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,259
  3. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 888
  4. 2024OLMo: Accelerating the Science of Language Modelscited by 599
  5. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 486
  6. 2023Benchmarking Large Language Models in Retrieval-Augmented Generationcited by 307
  7. 2024Spotting LLMs With Binoculars: Zero-Shot Detection of Machine-Generated Textcited by 255
  8. 2023The False Promise of Imitating Proprietary LLMscited by 254
  9. 2023ChatIE: Zero-Shot Information Extraction via Chatting with ChatGPTcited by 250
  10. 2023Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Taskscited by 243
  11. 2024On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Surveycited by 240
  12. 2023Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategiescited by 236
38 more
  1. 2024Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMscited by 224
  2. 2023On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspectivecited by 208
  3. 2023DetectLLM: Leveraging Log Rank Information for Zero-Shot Detection of Machine-Generated Textcited by 207
  4. 2023ChatGPT: Jack of all trades, master of nonecited by 204
  5. 2023ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learningcited by 203
  6. 2023Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedbackcited by 189
  7. 2023Can LLM-Generated Misinformation Be Detected?cited by 173
  8. 2023A Categorical Archive of ChatGPT Failurescited by 167
  9. 2023Personality Traits in Large Language Modelscited by 166
  10. 2023Is ChatGPT better than Human Annotators? Potential and Limitations of ChatGPT in Explaining Implicit Hate Speechcited by 165
  11. 2024RAID: A Shared Benchmark for Robust Evaluation of Machine-Generated Text Detectorscited by 157
  12. 2023Does Synthetic Data Generation of LLMs Help Clinical Text Mining?cited by 151
  13. 2023Evaluating ChatGPT's Information Extraction Capabilities: An Assessment of Performance, Explainability, Calibration, and Faithfulnesscited by 130
  14. 2023MAGE: Machine-generated Text Detection in the Wildcited by 130
  15. 2023The Science of Detecting LLM-Generated Textscited by 126
  16. 2023M4: Multi-generator, Multi-domain, and Multi-lingual Black-Box Machine-Generated Text Detectioncited by 118
  17. 2023Who Wrote this Code? Watermarking for Code Generationcited by 116
  18. 2023Intrinsic Dimension Estimation for Robust Detection of AI-Generated Textscited by 111
  19. 2023MoDS: Model-oriented Data Selection for Instruction Tuningcited by 107
  20. 2023A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directionscited by 101
  21. 2023Is ChatGPT a Good Sentiment Analyzer? A Preliminary Studycited by 96
  22. 2023A Survey of Safety and Trustworthiness of Large Language Models through the Lens of Verification and Validationcited by 91
  23. 2023MGTBench: Benchmarking Machine-Generated Text Detectioncited by 90
  24. 2023Ghostbuster: Detecting Text Ghostwritten by Large Language Modelscited by 90
  25. 2023Can ChatGPT Reproduce Human-Generated Labels? A Study of Social Computing Taskscited by 88
  26. 2023Testing of Detection Tools for AI-Generated Textcited by 86
  27. 2023OUTFOX: LLM-Generated Essay Detection Through In-Context Learning with Adversarially Generated Examplescited by 80
  28. 2023The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution in AIcited by 73
  29. 2024Detecting Multimedia Generated by Large AI Models: A Surveycited by 73
  30. 2023A Survey on Detection of LLMs-Generated Contentcited by 70
  31. 2023Let the LLMs Talk: Simulating Human-to-Human Conversational QA via Zero-Shot LLM-to-LLM Interactionscited by 70
  32. 2023Can Large Language Models Understand Real-World Complex Instructions?cited by 69
  33. 2023Watermarking Text Generated by Black-Box Language Modelscited by 68
  34. 2023Jais and Jais-chat: Arabic-Centric Foundation and Instruction-Tuned Open Generative Large Language Modelscited by 68
  35. 2024BRAVE: Broadening the visual encoding of vision-language modelscited by 68
  36. 2023FELM: Benchmarking Factuality Evaluation of Large Language Modelscited by 66
  37. 2023REMARK-LLM: A Robust and Efficient Watermarking Framework for Generative Large Language Modelscited by 64
  38. 2024(A)I Am Not a Lawyer, But...: Engaging Legal Experts towards Responsible LLM Policies for Legal Advicecited by 61

50 most cited of 472 All 472 (JSON)