# AI as Humanity's Salieri: Quantifying Linguistic Creativity of Language Models via Systematic Attribution of Machine Text against Web Text

arXiv: https://arxiv.org/abs/2410.04265 · HTML: https://mlsanity.com/paper/2410.04265/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 74 references that are in the map.
- First posted on arXiv 5 Oct 2024.

## Builds on (most cited first, 30 of 41)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization](https://mlsanity.com/paper/1808.08745/) (2018)
- [SpanBERT: Improving Pre-training by Representing and Predicting Spans](https://mlsanity.com/paper/1907.10529/) (2019)
- [DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature](https://mlsanity.com/paper/2301.11305/) (2023)
- [A Watermark for Large Language Models](https://mlsanity.com/paper/2301.10226/) (2023)
- [OLMo: Accelerating the Science of Language Models](https://mlsanity.com/paper/2402.00838/) (2024)
- [Release Strategies and the Social Impacts of Language Models](https://mlsanity.com/paper/1908.09203/) (2019)
- [GLTR: Statistical Detection and Visualization of Generated Text](https://mlsanity.com/paper/1906.04043/) (2019)
- [Detecting Pretraining Data from Large Language Models](https://mlsanity.com/paper/2310.16789/) (2023)
- [Fast-DetectGPT: Efficient Zero-Shot Detection of Machine-Generated Text via Conditional Probability Curvature](https://mlsanity.com/paper/2310.05130/) (2023)
- [Automatic Detection of Generated Text is Easiest when Humans are Fooled](https://mlsanity.com/paper/1911.00650/) (2019)
- [Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2](https://mlsanity.com/paper/2311.10702/) (2023)
- [Re3: Generating Longer Stories With Recursive Reprompting and Revision](https://mlsanity.com/paper/2210.06774/) (2022)
- [Co-Writing Screenplays and Theatre Scripts with Language Models: An Evaluation by Industry Professionals](https://mlsanity.com/paper/2209.14958/) (2022)
- [Art or Artifice? Large Language Models and the False Promise of Creativity](https://mlsanity.com/paper/2309.14556/) (2023)
- [Automatic Detection of Machine Generated Text: A Critical Survey](https://mlsanity.com/paper/2011.01314/) (2020)
- [Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokens](https://mlsanity.com/paper/2401.17377/) (2024)
- [Adversarial Watermarking Transformer: Towards Tracing Text Provenance with Data Hiding](https://mlsanity.com/paper/2009.03015/) (2020)
- [TURINGBENCH: A Benchmark Environment for Turing Test in the Age of Neural Text Generation](https://mlsanity.com/paper/2109.13296/) (2021)
- [Real or Fake? Learning to Discriminate Machine from Human Generated Text](https://mlsanity.com/paper/1906.03351/) (2019)
- [Ghostbuster: Detecting Text Ghostwritten by Large Language Models](https://mlsanity.com/paper/2305.15047/) (2023)
- [Neural Linguistic Steganography](https://mlsanity.com/paper/1909.01496/) (2019)
- [NaturalProver: Grounded Mathematical Proof Generation with Language Models](https://mlsanity.com/paper/2205.12910/) (2022)
- [Creative Writing with an AI-Powered Writing Assistant: Perspectives from Professional Writers](https://mlsanity.com/paper/2211.05030/) (2022)
- [Real or Fake Text?: Investigating Human Ability to Detect Boundaries Between Human-Written and Machine-Generated Text](https://mlsanity.com/paper/2212.12672/) (2022)
- [Generating Steganographic Text with LSTMs](https://mlsanity.com/paper/1705.10742/) (2017)

## Built on by (most cited first, 30 of 42)

- [ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models](https://mlsanity.com/paper/2505.24864/) (2025)
- [OverThink: Slowdown Attacks on Reasoning LLMs](https://mlsanity.com/paper/2502.02542/) (2025)
- [Jointly Reinforcing Diversity and Quality in Language Model Generations](https://mlsanity.com/paper/2509.02534/) (2025)
- [Base Models Beat Aligned Models at Randomness and Creativity](https://mlsanity.com/paper/2505.00047/) (2025)
- [Roll the dice & look before you leap: Going beyond the creative limits of next-token prediction](https://mlsanity.com/paper/2504.15266/) (2025)
- [LLM Probability Concentration: How Alignment Shrinks the Generative Horizon](https://mlsanity.com/paper/2506.17871/) (2025)
- [ProfBench: Multi-Domain Rubrics requiring Professional Knowledge to Answer and Judge](https://mlsanity.com/paper/2510.18941/) (2025)
- [All That Glitters is Not Novel: Plagiarism in AI Generated Research](https://mlsanity.com/paper/2502.16487/) (2025)
- [Measuring Non-Adversarial Reproduction of Training Data in Large Language Models](https://mlsanity.com/paper/2411.10242/) (2024)
- [Infini-gram mini: Exact n-gram Search at the Internet Scale with FM-Index](https://mlsanity.com/paper/2506.12229/) (2025)
- [QUDsim: Quantifying Discourse Similarities in LLM-Generated Text](https://mlsanity.com/paper/2504.09373/) (2025)
- [Blackbox Model Provenance via Palimpsestic Membership Inference](https://mlsanity.com/paper/2510.19796/) (2025)
- [Evaluating Text Creativity across Diverse Domains: A Dataset and Large Language Model Evaluator](https://mlsanity.com/paper/2505.19236/) (2025)
- [Frankentext: Stitching random text fragments into long-form narratives](https://mlsanity.com/paper/2505.18128/) (2025)
- [The Reader is the Metric: How Textual Features and Reader Profiles Explain Conflicting Evaluations of AI Creative Writing](https://mlsanity.com/paper/2506.03310/) (2025)
- [Optimizing Diversity and Quality through Base-Aligned Model Collaboration](https://mlsanity.com/paper/2511.05650/) (2025)
- [Creativity in AI: Progresses and Challenges](https://mlsanity.com/paper/2410.17218/) (2024)
- [The Surprising Effectiveness of Membership Inference with Simple N-Gram Coverage](https://mlsanity.com/paper/2508.09603/) (2025)
- [Measuring LLM Novelty As The Frontier Of Original And High-Quality Output](https://mlsanity.com/paper/2504.09389/) (2025)
- [ParaPO: Aligning Language Models to Reduce Verbatim Reproduction of Pre-training Data](https://mlsanity.com/paper/2504.14452/) (2025)
- [Death of the Novel(ty): Beyond n-Gram Novelty as a Metric for Textual Creativity](https://mlsanity.com/paper/2509.22641/) (2025)
- [Diagnosing Memorization in Chain-of-Thought Reasoning, One Token at a Time](https://mlsanity.com/paper/2508.02037/) (2025)
- [Societal Impacts Research Requires Benchmarks for Creative Composition Tasks](https://mlsanity.com/paper/2504.06549/) (2025)
- [Research Borderlands: Analysing Writing Across Research Cultures](https://mlsanity.com/paper/2506.00784/) (2025)
- [Beyond Divergent Creativity: A Human-Based Evaluation of Creativity in Large Language Models](https://mlsanity.com/paper/2601.20546/) (2026)
- [Enhancing Diversity of LLM-Generated Educational Tasks](https://mlsanity.com/paper/2512.23601/) (2025)
- [PowerFlow: Unlocking the Dual Nature of LLMs via Principled Distribution Matching](https://mlsanity.com/paper/2603.18363/) (2026)
- [Structure Before Collapse: Transient semantic geometry in next-token prediction](https://mlsanity.com/paper/2606.26749/) (2026)
- [Generative AI floods and dilutes the market for books](https://mlsanity.com/paper/2607.20349/) (2026)
- [Thinking Outside the (Gray) Box: A Context-Based Score for Assessing Value and Originality in Neural Text Generation](https://mlsanity.com/paper/2502.13207/) (2025)

## Abstract (arXiv)

Creativity has long been considered one of the most difficult aspect of human intelligence for AI to mimic. However, the rise of Large Language Models (LLMs), like ChatGPT, has raised questions about whether AI can match or even surpass human creativity. We present CREATIVITY INDEX as the first step to quantify the linguistic creativity of a text by reconstructing it from existing text snippets on the web. CREATIVITY INDEX is motivated by the hypothesis that the seemingly remarkable creativity of LLMs may be attributable in large part to the creativity of human-written texts on the web. To compute CREATIVITY INDEX efficiently, we introduce DJ SEARCH, a novel dynamic programming algorithm that can search verbatim and near-verbatim matches of text snippets from a given document against the web. Experiments reveal that the CREATIVITY INDEX of professional human authors is on average 66.2% higher than that of LLMs, and that alignment reduces the CREATIVITY INDEX of LLMs by an average of 30.1%. In addition, we find that distinguished authors like Hemingway exhibit measurably higher CREATIVITY INDEX compared to other human writers. Finally, we demonstrate that CREATIVITY INDEX can be used as a surprisingly effective criterion for zero-shot machine text detection, surpassing the strongest existing zero-shot system, DetectGPT, by a significant margin of 30.2%, and even outperforming the strongest supervised system, GhostBuster, in five out of six domains.