# ChatGPT vs Human-authored Text: Insights into Controllable Text Summarization and Sentence Style Transfer

arXiv: https://arxiv.org/abs/2306.07799 · HTML: https://mlsanity.com/paper/2306.07799/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- Builds on 50 of its 87 references that are in the map.
- First posted on arXiv 13 Jun 2023.

## Builds on (most cited first, 30 of 50)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Get To The Point: Summarization with Pointer-Generator Networks](https://mlsanity.com/paper/1704.04368/) (2017)
- [HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face](https://mlsanity.com/paper/2303.17580/) (2023)
- [Diffusion-LM Improves Controllable Text Generation](https://mlsanity.com/paper/2205.14217/) (2022)
- [A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity](https://mlsanity.com/paper/2302.04023/) (2023)
- [Plug and Play Language Models: A Simple Approach to Controlled Text Generation](https://mlsanity.com/paper/1912.02164/) (2019)
- [CTRL: A Conditional Transformer Language Model for Controllable Generation](https://mlsanity.com/paper/1909.05858/) (2019)
- [MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action](https://mlsanity.com/paper/2303.11381/) (2023)
- [Is ChatGPT a General-Purpose Natural Language Processing Task Solver?](https://mlsanity.com/paper/2302.06476/) (2023)
- [SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization](https://mlsanity.com/paper/2111.09525/) (2021)
- [A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPT](https://mlsanity.com/paper/2302.09419/) (2023)
- [Dear Sir or Madam, May I introduce the GYAFC Dataset: Corpus, Benchmarks and Metrics for Formality Style Transfer](https://mlsanity.com/paper/1803.06535/) (2018)
- [ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning](https://mlsanity.com/paper/2304.05613/) (2023)
- [Deep Learning for Text Style Transfer: A Survey](https://mlsanity.com/paper/2011.00416/) (2020)
- [Towards Making the Most of ChatGPT for Machine Translation](https://mlsanity.com/paper/2303.13780/) (2023)
- [Hallucinated but Factual! Inspecting the Factuality of Hallucinations in Abstractive Summarization](https://mlsanity.com/paper/2109.09784/) (2021)
- [A Survey of Controllable Text Generation using Transformer-based Pre-trained Language Models](https://mlsanity.com/paper/2201.05337/) (2022)
- [Evaluating Factuality in Generation with Dependency-level Entailment](https://mlsanity.com/paper/2010.05478/) (2020)
- [Exploring the Limits of ChatGPT for Query or Aspect-based Text Summarization](https://mlsanity.com/paper/2302.08081/) (2023)
- [Controllable Abstractive Summarization](https://mlsanity.com/paper/1711.05217/) (2017)
- [An Analysis of the Automatic Bug Fixing Performance of ChatGPT](https://mlsanity.com/paper/2301.08653/) (2023)
- [A Recipe For Arbitrary Text Style Transfer with Large Language Models](https://mlsanity.com/paper/2109.03910/) (2021)
- [CTRLsum: Towards Generic Controllable Text Summarization](https://mlsanity.com/paper/2012.04281/) (2020)
- [Style Transformer: Unpaired Text Style Transfer without Disentangled Latent Representation](https://mlsanity.com/paper/1905.05621/) (2019)
- [Gradient-Based Constrained Sampling from Language Models](https://mlsanity.com/paper/2205.12558/) (2022)
- [Evaluating Style Transfer for Text](https://mlsanity.com/paper/1904.02295/) (2019)
- [Making Science Simple: Corpora for the Lay Summarisation of Scientific Literature](https://mlsanity.com/paper/2210.09932/) (2022)
- [Aspect-Controllable Opinion Summarization](https://mlsanity.com/paper/2109.03171/) (2021)

## Built on by (most cited first, 30 of 50)

- [A Survey on Evaluation of Large Language Models](https://mlsanity.com/paper/2307.03109/) (2023)
- [Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategies](https://mlsanity.com/paper/2308.03188/) (2023)
- [Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMs](https://mlsanity.com/paper/2402.03927/) (2024)
- [Pearl: Personalizing Large Language Model Writing Assistants with Generation-Calibrated Retrievers](https://mlsanity.com/paper/2311.09180/) (2023)
- [Cognitive Mirage: A Review of Hallucinations in Large Language Models](https://mlsanity.com/paper/2309.06794/) (2023)
- [Benchmarking Generation and Evaluation Capabilities of Large Language Models for Instruction Controllable Summarization](https://mlsanity.com/paper/2311.09184/) (2023)
- [From Sparse to Dense: GPT-4 Summarization with Chain of Density Prompting](https://mlsanity.com/paper/2309.04269/) (2023)
- [A Systematic Survey of Text Summarization: From Statistical Methods to Large Language Models](https://mlsanity.com/paper/2406.11289/) (2024)
- [A Fast Optimization View: Reformulating Single Layer Attention in LLM Based on Tensor and SVM Trick, and Solving It in Matrix Multiplication Time](https://mlsanity.com/paper/2309.07418/) (2023)
- [FFT: Towards Harmlessness Evaluation and Analysis for LLMs with Factuality, Fairness, Toxicity](https://mlsanity.com/paper/2311.18580/) (2023)
- [Generating Summaries with Controllable Readability Levels](https://mlsanity.com/paper/2310.10623/) (2023)
- [Text Generation: A Systematic Literature Review of Tasks, Evaluation, and Challenges](https://mlsanity.com/paper/2405.15604/) (2024)
- [Flesch or Fumble? Evaluating Readability Standard Alignment of Instruction-Tuned Language Models](https://mlsanity.com/paper/2309.05454/) (2023)
- [Know the Unknown: An Uncertainty-Sensitive Method for LLM Instruction Tuning](https://mlsanity.com/paper/2406.10099/) (2024)
- [Generative Multimodal Entity Linking](https://mlsanity.com/paper/2306.12725/) (2023)
- [Distilling Text Style Transfer With Self-Explanation From LLMs](https://mlsanity.com/paper/2403.01106/) (2024)
- [SciNews: From Scholarly Complexities to Public Narratives -- A Dataset for Scientific News Report Generation](https://mlsanity.com/paper/2403.17768/) (2024)
- [RSA-Control: A Pragmatics-Grounded Lightweight Controllable Text Generation Framework](https://mlsanity.com/paper/2410.19109/) (2024)
- [What Is That Talk About? A Video-to-Text Summarization Dataset for Scientific Presentations](https://mlsanity.com/paper/2502.08279/) (2025)
- [DetoxLLM: A Framework for Detoxification with Explanations](https://mlsanity.com/paper/2402.15951/) (2024)
- [ReadCtrl: Personalizing text generation with readability-controlled instruction learning](https://mlsanity.com/paper/2406.09205/) (2024)
- [Measuring and Modifying the Readability of English Texts with GPT-4](https://mlsanity.com/paper/2410.14028/) (2024)
- [Beyond Profile: From Surface-Level Facts to Deep Persona Simulation in LLMs](https://mlsanity.com/paper/2502.12988/) (2025)
- [Computational Politeness in Natural Language Processing: A Survey](https://mlsanity.com/paper/2407.12814/) (2024)
- [PoetryDiffusion: Towards Joint Semantic and Metrical Manipulation in Poetry Generation](https://mlsanity.com/paper/2306.08456/) (2023)
- [Large Language Models are Interpretable Learners](https://mlsanity.com/paper/2406.17224/) (2024)
- [A Guide To Effectively Leveraging LLMs for Low-Resource Text Summarization: Data Augmentation and Semi-supervised Approaches](https://mlsanity.com/paper/2407.07341/) (2024)
- [Keystroke Dynamics Against Academic Dishonesty in the Age of LLMs](https://mlsanity.com/paper/2406.15335/) (2024)
- [An Evaluation of Large Language Models in Bioinformatics Research](https://mlsanity.com/paper/2402.13714/) (2024)
- [Standardizing Intelligence: Aligning Generative AI for Regulatory and Operational Compliance](https://mlsanity.com/paper/2503.04736/) (2025)

## Abstract (arXiv)

Large-scale language models, like ChatGPT, have garnered significant media attention and stunned the public with their remarkable capacity for generating coherent text from short natural language prompts. In this paper, we aim to conduct a systematic inspection of ChatGPT's performance in two controllable generation tasks, with respect to ChatGPT's ability to adapt its output to different target audiences (expert vs. layman) and writing styles (formal vs. informal). Additionally, we evaluate the faithfulness of the generated text, and compare the model's performance with human-authored texts. Our findings indicate that the stylistic variations produced by humans are considerably larger than those demonstrated by ChatGPT, and the generated texts diverge from human samples in several characteristics, such as the distribution of word types. Moreover, we observe that ChatGPT sometimes incorporates factual errors or hallucinations when adapting the text to suit a specific style.