# Using Machine Learning to Distinguish Human-written from Machine-generated Creative Fiction

arXiv: https://arxiv.org/abs/2412.15253 · HTML: https://mlsanity.com/paper/2412.15253/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 43 references that are in the map.
- First posted on arXiv 15 Dec 2024.

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature](https://mlsanity.com/paper/2301.11305/) (2023)
- [All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Text](https://mlsanity.com/paper/2107.00061/) (2021)
- [From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Models](https://mlsanity.com/paper/2305.08283/) (2023)
- [DetectLLM: Leveraging Log Rank Information for Zero-Shot Detection of Machine-Generated Text](https://mlsanity.com/paper/2306.05540/) (2023)
- [Automatic Detection of Machine Generated Text: A Critical Survey](https://mlsanity.com/paper/2011.01314/) (2020)
- [Ghostbuster: Detecting Text Ghostwritten by Large Language Models](https://mlsanity.com/paper/2305.15047/) (2023)
- [Machine Generated Text: A Comprehensive Survey of Threat Models and Detection Methods](https://mlsanity.com/paper/2210.07321/) (2022)
- [Creative Writing with an AI-Powered Writing Assistant: Perspectives from Professional Writers](https://mlsanity.com/paper/2211.05030/) (2022)
- [GPT-Sentinel: Distinguishing Human and ChatGPT Generated Content](https://mlsanity.com/paper/2305.07969/) (2023)
- [GPT-who: An Information Density-based Machine-Generated Text Detector](https://mlsanity.com/paper/2310.06202/) (2023)
- [Detecting LLM-Generated Text in Computing Education: A Comparative Study for ChatGPT Cases](https://mlsanity.com/paper/2307.07411/) (2023)
- [Distinguishing ChatGPT(-3.5, -4)-generated and human-written papers through Japanese stylometric analysis](https://mlsanity.com/paper/2304.05534/) (2023)
- [Exploring the Impact of AI Value Alignment in Collaborative Ideation: Effects on Perception, Ownership, and Output](https://mlsanity.com/paper/2402.12814/) (2024)
- [Distinguishing Human Generated Text From ChatGPT Generated Text Using Machine Learning](https://mlsanity.com/paper/2306.01761/) (2023)
- [Investigating Markers and Drivers of Gender Bias in Machine Translations](https://mlsanity.com/paper/2403.11896/) (2024)

## Built on by (most cited first, 4 of 4)

- [StoryScope: Investigating idiosyncrasies in AI fiction](https://mlsanity.com/paper/2604.03136/) (2026)
- [When Personalization Tricks Detectors: The Feature-Inversion Trap in Machine-Generated Text Detection](https://mlsanity.com/paper/2510.12476/) (2025)
- [Cat and Mouse -- Can Fake Text Generation Outpace Detector Systems?](https://mlsanity.com/paper/2506.21274/) (2025)
- [Detecting undisclosed LLM-generated content in parliamentary texts](https://mlsanity.com/paper/2606.14209/) (2026)

## Abstract (arXiv)

Following the universal availability of generative AI systems with the release of ChatGPT, automatic detection of deceptive text created by Large Language Models has focused on domains such as academic plagiarism and "fake news". However, generative AI also poses a threat to the livelihood of creative writers, and perhaps to literary culture in general, through reduction in quality of published material. Training a Large Language Model on writers' output to generate "sham books" in a particular style seems to constitute a new form of plagiarism. This problem has been little researched. In this study, we trained Machine Learning classifier models to distinguish short samples of human-written from machine-generated creative fiction, focusing on classic detective novels. Our results show that a Naive Bayes and a Multi-Layer Perceptron classifier achieved a high degree of success (accuracy > 95%), significantly outperforming human judges (accuracy < 55%). This approach worked well with short text samples (around 100 words), which previous research has shown to be difficult to classify. We have deployed an online proof-of-concept classifier tool, AI Detective, as a first step towards developing lightweight and reliable applications for use by editors and publishers, with the aim of protecting the economic and cultural contribution of human authors.