# Distinguishing ChatGPT(-3.5, -4)-generated and human-written papers through Japanese stylometric analysis

arXiv: https://arxiv.org/abs/2304.05534 · HTML: https://mlsanity.com/paper/2304.05534/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 12 references that are in the map.
- First posted on arXiv 11 Apr 2023.

## Builds on (most cited first, 4 of 4)

- [How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection](https://mlsanity.com/paper/2301.07597/) (2023)
- [All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Text](https://mlsanity.com/paper/2107.00061/) (2021)
- [ChatGPT or Human? Detect and Explain. Explaining Decisions of Machine Learning Model for Detecting Short ChatGPT-generated Text](https://mlsanity.com/paper/2301.13852/) (2023)
- [AI vs. Human -- Differentiation Analysis of Scientific Content Generation](https://mlsanity.com/paper/2301.10416/) (2023)

## Built on by (most cited first, 16 of 16)

- [Your Large Language Models Are Leaving Fingerprints](https://mlsanity.com/paper/2405.14057/) (2024)
- [Idiosyncrasies in Large Language Models](https://mlsanity.com/paper/2502.12150/) (2025)
- [Classification of Human- and AI-Generated Texts: Investigating Features for ChatGPT](https://mlsanity.com/paper/2308.05341/) (2023)
- [Linguistic Characteristics of AI-Generated Text: A Survey](https://mlsanity.com/paper/2510.05136/) (2025)
- [GenAI Content Detection Task 2: AI vs. Human -- Academic Essay Authenticity Challenge](https://mlsanity.com/paper/2412.18274/) (2024)
- [A Lightweight Approach to Detection of AI-Generated Texts Using Stylometric Features](https://mlsanity.com/paper/2511.21744/) (2025)
- [Classification of Human- and AI-Generated Texts for English, French, German, and Spanish](https://mlsanity.com/paper/2312.04882/) (2023)
- [Using Machine Learning to Distinguish Human-written from Machine-generated Creative Fiction](https://mlsanity.com/paper/2412.15253/) (2024)
- [Fine-tuning Large Language Models for Multigenerator, Multidomain, and Multilingual Machine-Generated Text Detection](https://mlsanity.com/paper/2401.12326/) (2024)
- [On the Detectability of LLM-Generated Text: What Exactly Is LLM-Generated Text?](https://mlsanity.com/paper/2510.20810/) (2025)
- [AI-Associated Lexical Shifts Across 34 Languages: Cross-Lingual Convergence and Diachronic Uptake in News Writing](https://mlsanity.com/paper/2605.25358/) (2026)
- [External Reasoning: Towards Multi-Large-Language-Models Interchangeable Assistance with Human Feedback](https://mlsanity.com/paper/2307.12057/) (2023)
- [Integrated ensemble of BERT- and features-based models for authorship attribution in Japanese literary works](https://mlsanity.com/paper/2504.08527/) (2025)
- [Missing the human touch? A computational stylometry analysis of GPT-4 translations of online Chinese literature](https://mlsanity.com/paper/2506.13013/) (2025)
- [Error-Aware Knowledge Distillation via Targeted Revision for Customer-Service Summarization](https://mlsanity.com/paper/2511.03005/) (2025)
- [Fully Automated Identification of Lexical Alignment and Preference-Stage Shifts in Large Language Models](https://mlsanity.com/paper/2606.03165/) (2026)

## Abstract (arXiv)

In the first half of 2023, text-generative artificial intelligence (AI), including ChatGPT, equipped with GPT-3.5 and GPT-4, from OpenAI, has attracted considerable attention worldwide. In this study, first, we compared Japanese stylometric features of texts generated by GPT (-3.5 and -4) and those written by humans. In this work, we performed multi-dimensional scaling (MDS) to confirm the distributions of 216 texts of three classes (72 academic papers written by 36 single authors, 72 texts generated by GPT-3.5, and 72 texts generated by GPT-4 on the basis of the titles of the aforementioned papers) focusing on the following stylometric features: (1) bigrams of parts-of-speech, (2) bigram of postpositional particle words, (3) positioning of commas, and (4) rate of function words. MDS revealed distinct distributions at each stylometric feature of GPT (-3.5 and -4) and human. Although GPT-4 is more powerful than GPT-3.5 because it has more parameters, both GPT (-3.5 and -4) distributions are likely to overlap. These results indicate that although the number of parameters may increase in the future, GPT-generated texts may not be close to that written by humans in terms of stylometric features. Second, we verified the classification performance of random forest (RF) for two classes (GPT and human) focusing on Japanese stylometric features. This study revealed the high performance of RF in each stylometric feature: The RF classifier focusing on the rate of function words achieved 98.1% accuracy. Furthermore the RF classifier focusing on all stylometric features reached 100% in terms of all performance indexes (accuracy, recall, precision, and F1 score). This study concluded that at this stage we human discriminate ChatGPT from human limited to Japanese language.