# Do LLMs write like humans? Variation in grammatical and rhetorical styles

arXiv: https://arxiv.org/abs/2410.16107 · HTML: https://mlsanity.com/paper/2410.16107/

## Key facts

- Cited by 62 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 52 references that are in the map.
- First posted on arXiv 21 Oct 2024.

## Builds on (most cited first, 14 of 14)

- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Towards Reasoning in Large Language Models: A Survey](https://mlsanity.com/paper/2212.10403/) (2022)
- [ranger: A Fast Implementation of Random Forests for High Dimensional Data in C++ and R](https://mlsanity.com/paper/1508.04409/) (2015)
- [Monitoring AI-Modified Content at Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviews](https://mlsanity.com/paper/2403.07183/) (2024)
- [DNA-GPT: Divergent N-Gram Analysis for Training-Free Detection of GPT-Generated Text](https://mlsanity.com/paper/2305.17359/) (2023)
- [The Science of Detecting LLM-Generated Texts](https://mlsanity.com/paper/2303.07205/) (2023)
- [M4: Multi-generator, Multi-domain, and Multi-lingual Black-Box Machine-Generated Text Detection](https://mlsanity.com/paper/2305.14902/) (2023)
- [Deception Abilities Emerged in Large Language Models](https://mlsanity.com/paper/2307.16513/) (2023)
- [Can ChatGPT Defend its Belief in Truth? Evaluating LLM Reasoning via Debate](https://mlsanity.com/paper/2305.13160/) (2023)
- [Language Model Behavior: A Comprehensive Survey](https://mlsanity.com/paper/2303.11504/) (2023)
- [Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey](https://mlsanity.com/paper/2210.07700/) (2022)
- [Contrasting Linguistic Patterns in Human and LLM-Generated News Text](https://mlsanity.com/paper/2308.09067/) (2023)
- [Instructed to Bias: Instruction-Tuned Language Models Exhibit Emergent Cognitive Bias](https://mlsanity.com/paper/2308.00225/) (2023)
- [ChatGPT as Research Scientist: Probing GPT's Capabilities as a Research Librarian, Research Ethicist, Data Generator and Data Predictor](https://mlsanity.com/paper/2406.14765/) (2024)

## Built on by (most cited first, 30 of 62)

- [Uncertainty Quantification in LLM Agents: Foundations, Emerging Challenges, and Opportunities](https://mlsanity.com/paper/2602.05073/) (2026)
- [Persona-Augmented Benchmarking: Evaluating LLMs Across Diverse Writing Styles](https://mlsanity.com/paper/2507.22168/) (2025)
- [Comparing LLM-generated and human-authored news text using formal syntactic theory](https://mlsanity.com/paper/2506.01407/) (2025)
- [A Comparison of DeepSeek and Other LLMs](https://mlsanity.com/paper/2502.03688/) (2025)
- [StoryScope: Investigating idiosyncrasies in AI fiction](https://mlsanity.com/paper/2604.03136/) (2026)
- [Benchmark of stylistic variation in LLM-generated texts](https://mlsanity.com/paper/2509.10179/) (2025)
- [Linguistic Characteristics of AI-Generated Text: A Survey](https://mlsanity.com/paper/2510.05136/) (2025)
- [Post-training makes large language models less human-like](https://mlsanity.com/paper/2605.07632/) (2026)
- [Wikipedia in the Era of LLMs: Evolution and Risks](https://mlsanity.com/paper/2503.02879/) (2025)
- [Evaluation Under Imperfect Benchmarks and Ratings: A Case Study in Text Simplification](https://mlsanity.com/paper/2504.09394/) (2025)
- [When Detection Fails: The Power of Fine-Tuned Models to Generate Human-Like Social Media Text](https://mlsanity.com/paper/2506.09975/) (2025)
- [PeerPrism: Peer Evaluation Expertise vs Review-writing AI](https://mlsanity.com/paper/2604.14513/) (2026)
- [Evaluating Style-Personalized Text Generation: Challenges and Directions](https://mlsanity.com/paper/2508.06374/) (2025)
- [On the Detectability of LLM-Generated Text: What Exactly Is LLM-Generated Text?](https://mlsanity.com/paper/2510.20810/) (2025)
- [Beyond the Final Actor: Modeling the Dual Roles of Creator and Editor for Fine-Grained LLM-Generated Text Detection](https://mlsanity.com/paper/2604.04932/) (2026)
- [From Safety Risk to Design Principle: Peer-Preservation in Multi-Agent LLM Systems and Its Implications for Orchestrated Democratic Discourse Analysis](https://mlsanity.com/paper/2604.08465/) (2026)
- [AI generates well-liked but templatic empathic responses](https://mlsanity.com/paper/2604.08479/) (2026)
- [Rhetorical Questions in LLM Representations: A Linear Probing Study](https://mlsanity.com/paper/2604.14128/) (2026)
- [More Aligned, Less Diverse? Analyzing the Grammar and Lexicon of Two Generations of LLMs](https://mlsanity.com/paper/2605.06030/) (2026)
- [ExpressivityBench: Can LLMs Communicate Implicitly?](https://mlsanity.com/paper/2411.08010/) (2024)
- [Beyond checkmate: exploring the creative chokepoints in AI text](https://mlsanity.com/paper/2501.19301/) (2025)
- [References Matter: Investigating the Impact of Reference Set Variation on Summarization Evaluation](https://mlsanity.com/paper/2506.14335/) (2025)
- [Better as Generators Than Classifiers: Leveraging LLMs and Synthetic Data for Low-Resource Multilingual Classification](https://mlsanity.com/paper/2601.16278/) (2026)
- [The Statistical Signature of LLMs](https://mlsanity.com/paper/2602.18152/) (2026)
- [Ethics and Persuasion in Reinforcement Learning from Human Feedback: A Procedural Rhetorical Approach](https://mlsanity.com/paper/2505.09576/) (2025)
- [V-VAE: A Variational Auto Encoding Framework Towards Fine-Grained Control over Human-Like Chat](https://mlsanity.com/paper/2506.01524/) (2025)
- [Do LLMs produce texts with "human-like" lexical diversity?](https://mlsanity.com/paper/2508.00086/) (2025)
- [Attribution Quality in AI-Generated Content:Benchmarking Style Embeddings and LLM Judges](https://mlsanity.com/paper/2510.13898/) (2025)
- [HLPD: Aligning LLMs to Human Language Preference for Machine-Revised Text Detection](https://mlsanity.com/paper/2511.06942/) (2025)
- [AudAgent: Automated Auditing of Privacy Policy Compliance in AI Agents](https://mlsanity.com/paper/2511.07441/) (2025)

## Abstract (arXiv)

Large language models (LLMs) are capable of writing grammatical text that follows instructions, answers questions, and solves problems. As they have advanced, it has become difficult to distinguish their output from human-written text. While past research has found some differences in surface features such as word choice and punctuation, and developed classifiers to detect LLM output, none has studied the rhetorical styles of LLMs. Using several variants of Llama 3 and GPT-4o, we construct two parallel corpora of human- and LLM-written texts from common prompts. Using Douglas Biber's set of lexical, grammatical, and rhetorical features, we identify systematic differences between LLMs and humans and between different LLMs. These differences persist when moving from smaller models to larger ones, and are larger for instruction-tuned models than base models. This observation of differences demonstrates that despite their advanced abilities, LLMs struggle to match human stylistic variation. Attention to more advanced linguistic features can hence detect patterns in their behavior not previously recognized.