mlsanityData to 7 Oct 2026

ChatGPT vs Human-authored Text: Insights into Controllable Text Summarization and Sentence Style Transfer

arXiv 2306.07799 ↗First posted 13 Jun 2023cs.CL cs.AI cs.LGcited by 49

Not on the map. Why?

Abstract (arXiv)

Large-scale language models, like ChatGPT, have garnered significant media attention and stunned the public with their remarkable capacity for generating coherent text from short natural language prompts. In this paper, we aim to conduct a systematic inspection of ChatGPT's performance in two controllable generation tasks, with respect to ChatGPT's ability to adapt its output to different target audiences (expert vs. layman) and writing styles (formal vs. informal). Additionally, we evaluate the faithfulness of the generated text, and compare the model's performance with human-authored texts. Our findings indicate that the stylistic variations produced by humans are considerably larger than those demonstrated by ChatGPT, and the generated texts diverge from human samples in several characteristics, such as the distribution of word types. Moreover, we observe that ChatGPT sometimes incorporates factual errors or hallucinations when adapting the text to suit a specific style.

Key facts. Cited by 49 papers in the map (data through 7 Oct 2026). Builds on 50 of its 87 references that are in the map. First posted on arXiv 13 Jun 2023.

Builds on

50 of 87 references.

References · 50

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,281
  5. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  6. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,006
  7. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 888
  8. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 793
  9. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 754
  10. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 511
  11. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 486
  12. 2021SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarizationcited by 402
38 more
  1. 2023A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPTcited by 364
  2. 2018Dear Sir or Madam, May I introduce the GYAFC Dataset: Corpus, Benchmarks and Metrics for Formality Style Transfercited by 209
  3. 2023ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learningcited by 203
  4. 2020Deep Learning for Text Style Transfer: A Surveycited by 161
  5. 2023Towards Making the Most of ChatGPT for Machine Translationcited by 138
  6. 2021Hallucinated but Factual! Inspecting the Factuality of Hallucinations in Abstractive Summarizationcited by 128
  7. 2022A Survey of Controllable Text Generation using Transformer-based Pre-trained Language Modelscited by 128
  8. 2020Evaluating Factuality in Generation with Dependency-level Entailmentcited by 122
  9. 2023Exploring the Limits of ChatGPT for Query or Aspect-based Text Summarizationcited by 122
  10. 2017Controllable Abstractive Summarizationcited by 110
  11. 2023An Analysis of the Automatic Bug Fixing Performance of ChatGPTcited by 108
  12. 2021A Recipe For Arbitrary Text Style Transfer with Large Language Modelscited by 107
  13. 2020CTRLsum: Towards Generic Controllable Text Summarizationcited by 98
  14. 2019Style Transformer: Unpaired Text Style Transfer without Disentangled Latent Representationcited by 84
  15. 2022Gradient-Based Constrained Sampling from Language Modelscited by 64
  16. 2019Evaluating Style Transfer for Textcited by 59
  17. 2022Making Science Simple: Corpora for the Lay Summarisation of Scientific Literaturecited by 54
  18. 2021Aspect-Controllable Opinion Summarizationcited by 47
  19. 2020Expertise Style Transfer: A New Task Towards Better Communication between Experts and Laymencited by 45
  20. 2022A Causal Lens for Controllable Text Generationcited by 45
  21. 2021MReD: A Meta-Review Dataset for Structure-Controllable Text Generationcited by 42
  22. 2021Learning Controllable Content Generatorscited by 40
  23. 2022A Distributional Lens for Multi-Aspect Controllable Text Generationcited by 33
  24. 2022DisCup: Discriminator Cooperative Unlikelihood Prompt-tuning for Controllable Text Generationcited by 30
  25. 2021Controllable Abstractive Dialogue Summarization with Sketch Supervisioncited by 27
  26. 2022Text Style Transfer for Bias Mitigation using Masked Language Modelingcited by 27
  27. 2020Multimodal Text Style Transfer for Outdoor Vision-and-Language Navigationcited by 22
  28. 2022Tailor: A Prompt-Based Approach to Attribute-Based Controlled Text Generationcited by 22
  29. 2021Topic-Guided Abstractive Multi-Document Summarizationcited by 19
  30. 2022Readability Controllable Biomedical Document Summarizationcited by 19
  31. 2021Controllable Summarization with Constrained Markov Decision Processcited by 17
  32. 2023Incorporating Distributions of Discourse Structure for Long Document Abstractive Summarizationcited by 17
  33. 2020Contextual Text Style Transfercited by 16
  34. 2022PSP: Pre-trained Soft Prompts for Few-Shot Abstractive Summarizationcited by 14
  35. 2023AI and the FCI: Can ChatGPT Project an Understanding of Introductory Physics?cited by 13
  36. 2022SentBS: Sentence-level Beam Search for Controllable Summarizationcited by 10
  37. 2022So Different Yet So Alike! Constrained Unsupervised Text Style Transfercited by 7
  38. 2021Syntax Matters! Syntax-Controlled in Text Style Transfercited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 15; 2024: 22; 2025: 7; 2026: 5.

49 papers.

Sort:

By area:

  1. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  2. 2023Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategiescited by 236
  3. 2024Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMscited by 224
  4. 2023Pearl: Personalizing Large Language Model Writing Assistants with Generation-Calibrated Retrieverscited by 91
  5. 2023Cognitive Mirage: A Review of Hallucinations in Large Language Modelscited by 83
  6. 2023Benchmarking Generation and Evaluation Capabilities of Large Language Models for Instruction Controllable Summarizationcited by 69
  7. 2023From Sparse to Dense: GPT-4 Summarization with Chain of Density Promptingcited by 52
  8. 2024A Systematic Survey of Text Summarization: From Statistical Methods to Large Language Modelscited by 52
  9. 2023A Fast Optimization View: Reformulating Single Layer Attention in LLM Based on Tensor and SVM Trick, and Solving It in Matrix Multiplication Timecited by 30
  10. 2023Generating Summaries with Controllable Readability Levelscited by 24
  11. 2023FFT: Towards Harmlessness Evaluation and Analysis for LLMs with Factuality, Fairness, Toxicitycited by 22
  12. 2024Text Generation: A Systematic Literature Review of Tasks, Evaluation, and Challengescited by 18
37 more
  1. 2023Flesch or Fumble? Evaluating Readability Standard Alignment of Instruction-Tuned Language Modelscited by 17
  2. 2024Know the Unknown: An Uncertainty-Sensitive Method for LLM Instruction Tuningcited by 16
  3. 2024Distilling Text Style Transfer With Self-Explanation From LLMscited by 12
  4. 2024SciNews: From Scholarly Complexities to Public Narratives -- A Dataset for Scientific News Report Generationcited by 12
  5. 2023Generative Multimodal Entity Linkingcited by 11
  6. 2024RSA-Control: A Pragmatics-Grounded Lightweight Controllable Text Generation Frameworkcited by 11
  7. 2025What Is That Talk About? A Video-to-Text Summarization Dataset for Scientific Presentationscited by 11
  8. 2024DetoxLLM: A Framework for Detoxification with Explanationscited by 10
  9. 2024ReadCtrl: Personalizing text generation with readability-controlled instruction learningcited by 10
  10. 2024Measuring and Modifying the Readability of English Texts with GPT-4cited by 10
  11. 2025Beyond Profile: From Surface-Level Facts to Deep Persona Simulation in LLMscited by 9
  12. 2024Computational Politeness in Natural Language Processing: A Surveycited by 8
  13. 2024Large Language Models are Interpretable Learnerscited by 7
  14. 2023PoetryDiffusion: Towards Joint Semantic and Metrical Manipulation in Poetry Generationcited by 6
  15. 2024Keystroke Dynamics Against Academic Dishonesty in the Age of LLMscited by 6
  16. 2024A Guide To Effectively Leveraging LLMs for Low-Resource Text Summarization: Data Augmentation and Semi-supervised Approachescited by 6
  17. 2024An Evaluation of Large Language Models in Bioinformatics Researchcited by 5
  18. 2025Standardizing Intelligence: Aligning Generative AI for Regulatory and Operational Compliancecited by 5
  19. 2024Do LLMs Implicitly Determine the Suitable Text Difficulty for Users?cited by 4
  20. 2025Explanatory Summarization with Discourse-Driven Planningcited by 4
  21. 2024Strategic Prompting for Conversational Tasks: A Comparative Analysis of Large Language Models Across Diverse Conversational Taskscited by 3
  22. 2025On the Detectability of LLM-Generated Text: What Exactly Is LLM-Generated Text?cited by 3
  23. 2023Using ChatGPT as a CAT tool in Easy Language translationcited by 2
  24. 2024ProSwitch: Knowledge-Guided Instruction Tuning to Switch Between Professional and Non-Professional Responsescited by 2
  25. 2025RephQA: Evaluating Readability of Large Language Models in Public Health Question Answeringcited by 2
  26. 2023ChatGPT and Persuasive Technologies for the Management and Delivery of Personalized Recommendations in Hotel Hospitalitycited by 1
  27. 2023Relation Extraction in underexplored biomedical domains: A diversity-optimised sampling and synthetic data generation approachcited by 1
  28. 2024Binary Hypothesis Testing for Softmax Models and Leverage Score Modelscited by 1
  29. 2025HLPD: Aligning LLMs to Human Language Preference for Machine-Revised Text Detectioncited by 1
  30. 2026Disco-RAG: Discourse-Aware Retrieval-Augmented Generationcited by 1
  31. 2024A Survey on the Real Power of ChatGPTcited by 0
  32. 2024Human Speech Perception in Noise: Can Large Language Models Paraphrase to Improve It?cited by 0
  33. 2024FiSTECH: Financial Style Transfer to Enhance Creativity without Hallucinations in LLMscited by 0
  34. 2026Improvisational Games as a Benchmark for Social Intelligence of AI Agents: The Case of Connectionscited by 0
  35. 2026A Comparative Study of Controlled Text Generation Systems Using Level-Playing-Field Evaluation Principlescited by 0
  36. 2026A Multi-Model Metric-based Selection Framework for Abstractive Text summarizationcited by 0
  37. 2026Summarization is Not Dead Yetcited by 0