mlsanityData to 14 Oct 2026

Measuring AI "Slop" in Text

arXiv 2509.19163 ↗First posted 23 Sep 2025cs.CLcited by 6

Not on the map. Why?

Abstract (arXiv)

AI "slop" is an increasingly popular term used to describe low-quality AI-generated text, but there is currently no agreed upon definition of this term nor a means to measure its occurrence. In this work, we develop a taxonomy of "slop" through interviews with experts in NLP, writing, and philosophy, and propose a set of interpretable dimensions for its assessment in text. Through span-level annotation, we find that binary "slop" judgments are (somewhat) subjective, but such determinations nonetheless correlate with latent dimensions such as coherence and relevance. Our framework can be used to evaluate AI-generated text in both detection and binary preference tasks, potentially offering new insights into the linguistic and stylistic factors that contribute to quality judgments.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 28 of its 57 references that are in the map. First posted on arXiv 23 Sep 2025.

Builds on

28 of 57 references.

References · 28

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  3. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,605
  4. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 956
  5. 2023DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvaturecited by 726
  6. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 465
  7. 2024Spotting LLMs With Binoculars: Zero-Shot Detection of Machine-Generated Textcited by 263
  8. 2023Does Writing with Language Models Reduce Content Diversity?cited by 192
  9. 2023Art or Artifice? Large Language Models and the False Promise of Creativitycited by 172
  10. 2020Evaluating the Evaluation of Diversity in Natural Language Generationcited by 129
  11. 2025SycEval: Evaluating LLM Sycophancycited by 108
  12. 2025Which Economic Tasks are Performed with AI? Evidence from Millions of Claude Conversationscited by 96
16 more
  1. 2021Revisiting the Uniform Information Density Hypothesiscited by 68
  2. 2024Standardizing the Measurement of Text Diversity: A Tool and a Comparative Analysis of Scorescited by 50
  3. 2024Can AI writing be salvaged? Mitigating Idiosyncrasies and Improving Human-AI Alignment in the Writing Process through Editscited by 49
  4. 2024Detection and Measurement of Syntactic Templates in Generated Textcited by 44
  5. 2025People who frequently use ChatGPT for writing tasks are accurate and robust detectors of AI-generated textcited by 40
  6. 2024Evaluating the Evaluator: Measuring LLMs' Adherence to Task Evaluation Instructionscited by 34
  7. 2024LLM-based relevance assessment still can't replace human relevance assessmentcited by 27
  8. 2025AI-Slop to AI-Polish? Aligning Language Models through Edit-Based Writing Rewards and Test-time Computationcited by 25
  9. 2023Are Large Language Models Reliable Judges? A Study on the Factuality Evaluation Capabilities of LLMscited by 24
  10. 2024Can We Trust LLMs? Mitigate Overconfidence Bias in LLMs through Knowledge Transfercited by 19
  11. 2022Self-Repetition in Abstractive Neural Summarizerscited by 13
  12. 2025QUDsim: Quantifying Discourse Similarities in LLM-Generated Textcited by 13
  13. 2025Flattery, Fluff, and Fog: Diagnosing and Mitigating Idiosyncratic Biases in Preference Modelscited by 13
  14. 2024Do Automatic Factuality Metrics Measure Factuality? A Critical Evaluationcited by 11
  15. 2024Evaluating the Factuality of Zero-shot Summarizers Across Varied Domainscited by 10
  16. 2025Writing as a testbed for open ended agentscited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 4.

6 papers.

  1. 2025Death of the Novel(ty): Beyond n-Gram Novelty as a Metric for Textual Creativitycited by 5
  2. 2026AI generates well-liked but templatic empathic responsescited by 3
  3. 2026The Limits of Learning from Pictures and Text: Vision-Language Models and Embodied Scene Understandingcited by 2
  4. 2026Generative AI floods and dilutes the market for bookscited by 2
  5. 2025Surface Reading LLMs: Synthetic Text and its Stylescited by 0
  6. 2026TRIM: Reducing AI-Generated CodeSlop via Agent Trajectory Minimizationcited by 0