mlsanityData to 7 Oct 2026

AI as Humanity's Salieri: Quantifying Linguistic Creativity of Language Models via Systematic Attribution of Machine Text against Web Text

arXiv 2410.04265 ↗First posted 5 Oct 2024cs.CLcited by 40

Not on the map. Why?

Abstract (arXiv)

Creativity has long been considered one of the most difficult aspect of human intelligence for AI to mimic. However, the rise of Large Language Models (LLMs), like ChatGPT, has raised questions about whether AI can match or even surpass human creativity. We present CREATIVITY INDEX as the first step to quantify the linguistic creativity of a text by reconstructing it from existing text snippets on the web. CREATIVITY INDEX is motivated by the hypothesis that the seemingly remarkable creativity of LLMs may be attributable in large part to the creativity of human-written texts on the web. To compute CREATIVITY INDEX efficiently, we introduce DJ SEARCH, a novel dynamic programming algorithm that can search verbatim and near-verbatim matches of text snippets from a given document against the web. Experiments reveal that the CREATIVITY INDEX of professional human authors is on average 66.2% higher than that of LLMs, and that alignment reduces the CREATIVITY INDEX of LLMs by an average of 30.1%. In addition, we find that distinguished authors like Hemingway exhibit measurably higher CREATIVITY INDEX compared to other human writers. Finally, we demonstrate that CREATIVITY INDEX can be used as a surprisingly effective criterion for zero-shot machine text detection, surpassing the strongest existing zero-shot system, DetectGPT, by a significant margin of 30.2%, and even outperforming the strongest supervised system, GhostBuster, in five out of six domains.

Key facts. Cited by 40 papers in the map (data through 7 Oct 2026). Builds on 41 of its 74 references that are in the map. First posted on arXiv 5 Oct 2024.

Builds on

41 of 74 references.

References · 41

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  7. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 832
  8. 2023DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvaturecited by 705
  9. 2023A Watermark for Large Language Modelscited by 667
  10. 2024OLMo: Accelerating the Science of Language Modelscited by 599
  11. 2019Release Strategies and the Social Impacts of Language Modelscited by 505
  12. 2019GLTR: Statistical Detection and Visualization of Generated Textcited by 490
29 more
  1. 2023Detecting Pretraining Data from Large Language Modelscited by 375
  2. 2023Fast-DetectGPT: Efficient Zero-Shot Detection of Machine-Generated Text via Conditional Probability Curvaturecited by 278
  3. 2019Automatic Detection of Generated Text is Easiest when Humans are Fooledcited by 260
  4. 2023Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2cited by 227
  5. 2022Re3: Generating Longer Stories With Recursive Reprompting and Revisioncited by 193
  6. 2022Co-Writing Screenplays and Theatre Scripts with Language Models: An Evaluation by Industry Professionalscited by 191
  7. 2023Art or Artifice? Large Language Models and the False Promise of Creativitycited by 168
  8. 2020Automatic Detection of Machine Generated Text: A Critical Surveycited by 160
  9. 2024Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokenscited by 148
  10. 2020Adversarial Watermarking Transformer: Towards Tracing Text Provenance with Data Hidingcited by 121
  11. 2021TURINGBENCH: A Benchmark Environment for Turing Test in the Age of Neural Text Generationcited by 118
  12. 2019Real or Fake? Learning to Discriminate Machine from Human Generated Textcited by 98
  13. 2023Ghostbuster: Detecting Text Ghostwritten by Large Language Modelscited by 90
  14. 2019Neural Linguistic Steganographycited by 73
  15. 2022NaturalProver: Grounded Mathematical Proof Generation with Language Modelscited by 72
  16. 2022Creative Writing with an AI-Powered Writing Assistant: Perspectives from Professional Writerscited by 69
  17. 2022Real or Fake Text?: Investigating Human Ability to Detect Boundaries Between Human-Written and Machine-Generated Textcited by 63
  18. 2017Generating Steganographic Text with LSTMscited by 41
  19. 2024ComposerX: Multi-Agent Symbolic Music Composition with LLMscited by 40
  20. 2021Misinfo Reaction Frames: Reasoning about Readers' Reactions to News Headlinescited by 32
  21. 2023Creativity Support in the Age of Large Language Models: An Empirical Study Involving Emerging Writerscited by 32
  22. 2023Bot or Human? Detecting ChatGPT Imposters with A Single Questioncited by 25
  23. 2024Evaluating n-Gram Novelty of Language Models Using Rusty-DAWGcited by 24
  24. 2024Understanding Transformers via N-gram Statisticscited by 23
  25. 2024Large Language Model-Enabled Multi-Agent Manufacturing Systemscited by 14
  26. 2023Bits of Grass: Does GPT already know how to write like Whitman?cited by 13
  27. 2023Towards Foundational AI Models for Additive Manufacturing: Language Models for G-Code Debugging, Manipulation, and Comprehensioncited by 9
  28. 2024Can AI Write Classical Chinese Poetry like Humans? An Empirical Study Inspired by Turing Testcited by 7
  29. 2024ByteComposer: a Human-like Melody Composition Method based on Language Model Agentcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 26; 2026: 12.

40 papers.

Sort:

By area:

  1. 2025ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Modelscited by 187
  2. 2025OverThink: Slowdown Attacks on Reasoning LLMscited by 94
  3. 2025Jointly Reinforcing Diversity and Quality in Language Model Generationscited by 79
  4. 2025Base Models Beat Aligned Models at Randomness and Creativitycited by 48
  5. 2025Roll the dice & look before you leap: Going beyond the creative limits of next-token predictioncited by 28
  6. 2025LLM Probability Concentration: How Alignment Shrinks the Generative Horizoncited by 26
  7. 2025ProfBench: Multi-Domain Rubrics requiring Professional Knowledge to Answer and Judgecited by 21
  8. 2025All That Glitters is Not Novel: Plagiarism in AI Generated Researchcited by 17
  9. 2024Measuring Non-Adversarial Reproduction of Training Data in Large Language Modelscited by 14
  10. 2025QUDsim: Quantifying Discourse Similarities in LLM-Generated Textcited by 13
  11. 2025Infini-gram mini: Exact n-gram Search at the Internet Scale with FM-Indexcited by 13
  12. 2025Blackbox Model Provenance via Palimpsestic Membership Inferencecited by 11
28 more
  1. 2025Evaluating Text Creativity across Diverse Domains: A Dataset and Large Language Model Evaluatorcited by 9
  2. 2025Frankentext: Stitching random text fragments into long-form narrativescited by 8
  3. 2025Optimizing Diversity and Quality through Base-Aligned Model Collaborationcited by 8
  4. 2024Creativity in AI: Progresses and Challengescited by 7
  5. 2025The Reader is the Metric: How Textual Features and Reader Profiles Explain Conflicting Evaluations of AI Creative Writingcited by 7
  6. 2025The Surprising Effectiveness of Membership Inference with Simple N-Gram Coveragecited by 7
  7. 2025ParaPO: Aligning Language Models to Reduce Verbatim Reproduction of Pre-training Datacited by 6
  8. 2025Measuring LLM Novelty As The Frontier Of Original And High-Quality Outputcited by 5
  9. 2025Diagnosing Memorization in Chain-of-Thought Reasoning, One Token at a Timecited by 4
  10. 2025Death of the Novel(ty): Beyond n-Gram Novelty as a Metric for Textual Creativitycited by 4
  11. 2025Societal Impacts Research Requires Benchmarks for Creative Composition Taskscited by 3
  12. 2025Research Borderlands: Analysing Writing Across Research Culturescited by 3
  13. 2026Beyond Divergent Creativity: A Human-Based Evaluation of Creativity in Large Language Modelscited by 3
  14. 2025Enhancing Diversity of LLM-Generated Educational Taskscited by 2
  15. 2026PowerFlow: Unlocking the Dual Nature of LLMs via Principled Distribution Matchingcited by 2
  16. 2026Structure Before Collapse: Transient semantic geometry in next-token predictioncited by 2
  17. 2026Generative AI floods and dilutes the market for bookscited by 2
  18. 2025Thinking Outside the (Gray) Box: A Context-Based Score for Assessing Value and Originality in Neural Text Generationcited by 1
  19. 2025GuessBench: Sensemaking Multimodal Creativity in the Wildcited by 1
  20. 2026Ex Ante Evaluation of AI-Induced Idea Diversity Collapsecited by 1
  21. 2026Do Large Language Models Always Tell The Same Stories?cited by 1
  22. 2026The Limits of Automatic Evaluation of Creativity in Large Language Modelscited by 1
  23. 2025Leveraging Large Models to Evaluate Novel Content: A Case Study on Advertisement Creativitycited by 0
  24. 2026Retell, Reward, Repeat: Reinforcement Learning for Narrative Theory-Informed Story Retellingcited by 0
  25. 2026CresOWLve: Benchmarking Creative Problem-Solving Over Real-World Knowledgecited by 0
  26. 2026Beneath the Surface: Investigating LLMs' Capabilities for Communicating with Subtextcited by 0
  27. 2026How Creative Are Large Language Models in Generating Molecules?cited by 0
  28. 2026Constrained Semantic Decompression in LLMs through Persian Proverb-Conditioned Story Generationcited by 0