mlsanityData to 14 Oct 2026

Comparing Abstractive Summaries Generated by ChatGPT to Real Summaries Through Blinded Reviewers and Text Classification Algorithms

arXiv 2303.17650 ↗First posted 30 Mar 2023cs.CLcited by 15

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) have gathered significant attention due to their impressive performance on a variety of tasks. ChatGPT, developed by OpenAI, is a recent addition to the family of language models and is being called a disruptive technology by a few, owing to its human-like text-generation capabilities. Although, many anecdotal examples across the internet have evaluated ChatGPT's strength and weakness, only a few systematic research studies exist. To contribute to the body of literature of systematic research on ChatGPT, we evaluate the performance of ChatGPT on Abstractive Summarization by the means of automated metrics and blinded human reviewers. We also build automatic text classifiers to detect ChatGPT generated summaries. We found that while text classification algorithms can distinguish between real and generated summaries, humans are unable to distinguish between real summaries and those produced by ChatGPT.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Builds on 15 of its 28 references that are in the map. First posted on arXiv 30 Mar 2023.

Builds on

15 of 28 references.

References · 15

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  7. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,166
  8. 2017Deep reinforcement learning from human preferencescited by 3,912
  9. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  10. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,292
  11. 2015Teaching Machines to Read and Comprehendcited by 1,049
  12. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 894
3 more
  1. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 812
  2. 2015A Neural Attention Model for Abstractive Sentence Summarizationcited by 396
  3. 2023Mathematical Capabilities of ChatGPTcited by 268

Built on by

Cited by year

Citing papers per year: 2023: 7; 2024: 5; 2025: 3.

15 papers.

Sort:

  1. 2024Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMscited by 229
  2. 2023GPTEval: A Survey on Assessments of ChatGPT and GPT-4cited by 73
  3. 2023Disinformation Detection: An Evolving Challenge in the Age of LLMscited by 41
  4. 2023Classification of Human- and AI-Generated Texts: Investigating Features for ChatGPTcited by 19
  5. 2025DetectAnyLLM: Towards Generalizable and Robust Detection of Machine-Generated Text Across Domains and Modelscited by 9
  6. 2023Cross-lingual Cross-temporal Summarization: Dataset, Models, Evaluationcited by 8
  7. 2024ESPERANTO: Evaluating Synthesized Phrases to Enhance Robustness in AI Detection for Text Originationcited by 6
  8. 2023Hybrid Long Document Summarization using C2F-FAR and ChatGPT: A Practical Studycited by 5
  9. 2025Paladin: Defending LLM-enabled Phishing Emails with a New Trigger-Tag Paradigmcited by 5
  10. 2023Classification of Human- and AI-Generated Texts for English, French, German, and Spanishcited by 4
  11. 2025Advances in Pre-trained Language Models for Domain-Specific Text Classification: A Systematic Reviewcited by 3
  12. 2024Exploring ChatGPT for Next-generation Information Retrieval: Opportunities and Challengescited by 2
3 more
  1. 2023A study of the impact of generative AI-based data augmentation on software metadata classificationcited by 0
  2. 2024Research on Information Extraction of LCSTS Dataset Based on an Improved BERTSum-LSTM Modelcited by 0
  3. 2024DISCOVER: A Data-driven Interactive System for Comprehensive Observation, Visualization, and ExploRation of Human Behaviourcited by 0