# Stylometry recognizes human and LLM-generated texts in short samples

arXiv: https://arxiv.org/abs/2507.00838 · HTML: https://mlsanity.com/paper/2507.00838/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 46 of its 117 references that are in the map.
- First posted on arXiv 1 Jul 2025.

## Builds on (most cited first, 30 of 46)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [Defending Against Neural Fake News](https://mlsanity.com/paper/1905.12616/) (2019)
- [The Falcon Series of Open Language Models](https://mlsanity.com/paper/2311.16867/) (2023)
- [Datasets: A Community Library for Natural Language Processing](https://mlsanity.com/paper/2109.02846/) (2021)
- [How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection](https://mlsanity.com/paper/2301.07597/) (2023)
- [A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly](https://mlsanity.com/paper/2312.02003/) (2023)
- [Orca: Progressive Learning from Complex Explanation Traces of GPT-4](https://mlsanity.com/paper/2306.02707/) (2023)
- [Open-Set Recognition: a Good Closed-Set Classifier is All You Need?](https://mlsanity.com/paper/2110.06207/) (2021)
- [Recent Advances in Open Set Recognition: A Survey](https://mlsanity.com/paper/1811.08581/) (2018)
- [Deep Facial Expression Recognition: A Survey](https://mlsanity.com/paper/1804.08348/) (2018)
- [MAGE: Machine-generated Text Detection in the Wild](https://mlsanity.com/paper/2305.13242/) (2023)
- [M4: Multi-generator, Multi-domain, and Multi-lingual Black-Box Machine-Generated Text Detection](https://mlsanity.com/paper/2305.14902/) (2023)
- [TURINGBENCH: A Benchmark Environment for Turing Test in the Age of Neural Text Generation](https://mlsanity.com/paper/2109.13296/) (2021)
- [TweepFake: about Detecting Deepfake Tweets](https://mlsanity.com/paper/2008.00036/) (2020)
- [A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions](https://mlsanity.com/paper/2310.14724/) (2023)
- [A Survey of Safety and Trustworthiness of Large Language Models through the Lens of Verification and Validation](https://mlsanity.com/paper/2305.11391/) (2023)
- [Machine Generated Text: A Comprehensive Survey of Threat Models and Detection Methods](https://mlsanity.com/paper/2210.07321/) (2022)
- [Evaluating Large Language Model Biases in Persona-Steered Generation](https://mlsanity.com/paper/2405.20253/) (2024)
- [End-to-End Multimodal Emotion Recognition using Deep Neural Networks](https://mlsanity.com/paper/1704.08619/) (2017)
- [The Limitations of Stylometry for Detecting Machine-Generated Fake News](https://mlsanity.com/paper/1908.09805/) (2019)
- [Stylometric Detection of AI-Generated Text in Twitter Timelines](https://mlsanity.com/paper/2303.03697/) (2023)
- [ChatGPT and a New Academic Reality: Artificial Intelligence-Written Research Papers and the Ethics of the Large Language Models in Scholarly Publishing](https://mlsanity.com/paper/2303.13367/) (2023)
- [Large Language Models can be Guided to Evade AI-Generated Text Detection](https://mlsanity.com/paper/2305.10847/) (2023)
- [ArguGPT: evaluating, understanding and identifying argumentative essays generated by GPT models](https://mlsanity.com/paper/2304.07666/) (2023)
- [Can Large Language Models Identify Authorship?](https://mlsanity.com/paper/2403.08213/) (2024)
- [CHEAT: A Large-scale Dataset for Detecting ChatGPT-writtEn AbsTracts](https://mlsanity.com/paper/2304.12008/) (2023)
- [ConDA: Contrastive Domain Adaptation for AI-generated Text Detection](https://mlsanity.com/paper/2309.03992/) (2023)
- [HC3 Plus: A Semantic-Invariant Human ChatGPT Comparison Corpus](https://mlsanity.com/paper/2309.02731/) (2023)

## Abstract (arXiv)

The paper explores stylometry as a method to distinguish between texts created by Large Language Models (LLMs) and humans, addressing issues of model attribution, intellectual property, and ethical AI use. Stylometry has been used extensively to characterise the style and attribute authorship of texts. By applying it to LLM-generated texts, we identify their emergent writing patterns. The paper involves creating a benchmark dataset based on Wikipedia, with (a) human-written term summaries, (b) texts generated purely by LLMs (GPT-3.5/4, LLaMa 2/3, Orca, and Falcon), (c) processed through multiple text summarisation methods (T5, BART, Gensim, and Sumy), and (d) rephrasing methods (Dipper, T5). The 10-sentence long texts were classified by tree-based models (decision trees and LightGBM) using human-designed (StyloMetrix) and n-gram-based (our own pipeline) stylometric features that encode lexical, grammatical, syntactic, and punctuation patterns. The cross-validated results reached a performance of up to .87 Matthews correlation coefficient in the multiclass scenario with 7 classes, and accuracy between .79 and 1. in binary classification, with the particular example of Wikipedia and GPT-4 reaching up to .98 accuracy on a balanced dataset. Shapley Additive Explanations pinpointed features characteristic of the encyclopaedic text type, individual overused words, as well as a greater grammatical standardisation of LLMs with respect to human-written texts. These results show -- crucially, in the context of the increasingly sophisticated LLMs -- that it is possible to distinguish machine- from human-generated texts at least for a well-defined text type.