# Comparing Abstractive Summaries Generated by ChatGPT to Real Summaries Through Blinded Reviewers and Text Classification Algorithms

arXiv: https://arxiv.org/abs/2303.17650 · HTML: https://mlsanity.com/paper/2303.17650/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 28 references that are in the map.
- First posted on arXiv 30 Mar 2023.

## Builds on (most cited first, 15 of 15)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [Get To The Point: Summarization with Pointer-Generator Networks](https://mlsanity.com/paper/1704.04368/) (2017)
- [Teaching Machines to Read and Comprehend](https://mlsanity.com/paper/1506.03340/) (2015)
- [A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity](https://mlsanity.com/paper/2302.04023/) (2023)
- [Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond](https://mlsanity.com/paper/1602.06023/) (2016)
- [A Neural Attention Model for Abstractive Sentence Summarization](https://mlsanity.com/paper/1509.00685/) (2015)
- [Mathematical Capabilities of ChatGPT](https://mlsanity.com/paper/2301.13867/) (2023)

## Built on by (most cited first, 15 of 15)

- [Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMs](https://mlsanity.com/paper/2402.03927/) (2024)
- [GPTEval: A Survey on Assessments of ChatGPT and GPT-4](https://mlsanity.com/paper/2308.12488/) (2023)
- [Disinformation Detection: An Evolving Challenge in the Age of LLMs](https://mlsanity.com/paper/2309.15847/) (2023)
- [Classification of Human- and AI-Generated Texts: Investigating Features for ChatGPT](https://mlsanity.com/paper/2308.05341/) (2023)
- [DetectAnyLLM: Towards Generalizable and Robust Detection of Machine-Generated Text Across Domains and Models](https://mlsanity.com/paper/2509.14268/) (2025)
- [Cross-lingual Cross-temporal Summarization: Dataset, Models, Evaluation](https://mlsanity.com/paper/2306.12916/) (2023)
- [ESPERANTO: Evaluating Synthesized Phrases to Enhance Robustness in AI Detection for Text Origination](https://mlsanity.com/paper/2409.14285/) (2024)
- [Hybrid Long Document Summarization using C2F-FAR and ChatGPT: A Practical Study](https://mlsanity.com/paper/2306.01169/) (2023)
- [Paladin: Defending LLM-enabled Phishing Emails with a New Trigger-Tag Paradigm](https://mlsanity.com/paper/2509.07287/) (2025)
- [Classification of Human- and AI-Generated Texts for English, French, German, and Spanish](https://mlsanity.com/paper/2312.04882/) (2023)
- [Advances in Pre-trained Language Models for Domain-Specific Text Classification: A Systematic Review](https://mlsanity.com/paper/2510.17892/) (2025)
- [Exploring ChatGPT for Next-generation Information Retrieval: Opportunities and Challenges](https://mlsanity.com/paper/2402.11203/) (2024)
- [A study of the impact of generative AI-based data augmentation on software metadata classification](https://mlsanity.com/paper/2310.13714/) (2023)
- [Research on Information Extraction of LCSTS Dataset Based on an Improved BERTSum-LSTM Model](https://mlsanity.com/paper/2406.18364/) (2024)
- [DISCOVER: A Data-driven Interactive System for Comprehensive Observation, Visualization, and ExploRation of Human Behaviour](https://mlsanity.com/paper/2407.13408/) (2024)

## Abstract (arXiv)

Large Language Models (LLMs) have gathered significant attention due to their impressive performance on a variety of tasks. ChatGPT, developed by OpenAI, is a recent addition to the family of language models and is being called a disruptive technology by a few, owing to its human-like text-generation capabilities. Although, many anecdotal examples across the internet have evaluated ChatGPT's strength and weakness, only a few systematic research studies exist. To contribute to the body of literature of systematic research on ChatGPT, we evaluate the performance of ChatGPT on Abstractive Summarization by the means of automated metrics and blinded human reviewers. We also build automatic text classifiers to detect ChatGPT generated summaries. We found that while text classification algorithms can distinguish between real and generated summaries, humans are unable to distinguish between real summaries and those produced by ChatGPT.