# Measuring AI "Slop" in Text

arXiv: https://arxiv.org/abs/2509.19163 · HTML: https://mlsanity.com/paper/2509.19163/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 57 references that are in the map.
- First posted on arXiv 23 Sep 2025.

## Builds on (most cited first, 28 of 28)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [Language (Technology) is Power: A Critical Survey of "Bias" in NLP](https://mlsanity.com/paper/2005.14050/) (2020)
- [DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature](https://mlsanity.com/paper/2301.11305/) (2023)
- [LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods](https://mlsanity.com/paper/2412.05579/) (2024)
- [Spotting LLMs With Binoculars: Zero-Shot Detection of Machine-Generated Text](https://mlsanity.com/paper/2401.12070/) (2024)
- [Does Writing with Language Models Reduce Content Diversity?](https://mlsanity.com/paper/2309.05196/) (2023)
- [Art or Artifice? Large Language Models and the False Promise of Creativity](https://mlsanity.com/paper/2309.14556/) (2023)
- [Evaluating the Evaluation of Diversity in Natural Language Generation](https://mlsanity.com/paper/2004.02990/) (2020)
- [SycEval: Evaluating LLM Sycophancy](https://mlsanity.com/paper/2502.08177/) (2025)
- [Which Economic Tasks are Performed with AI? Evidence from Millions of Claude Conversations](https://mlsanity.com/paper/2503.04761/) (2025)
- [Revisiting the Uniform Information Density Hypothesis](https://mlsanity.com/paper/2109.11635/) (2021)
- [Standardizing the Measurement of Text Diversity: A Tool and a Comparative Analysis of Scores](https://mlsanity.com/paper/2403.00553/) (2024)
- [Can AI writing be salvaged? Mitigating Idiosyncrasies and Improving Human-AI Alignment in the Writing Process through Edits](https://mlsanity.com/paper/2409.14509/) (2024)
- [Detection and Measurement of Syntactic Templates in Generated Text](https://mlsanity.com/paper/2407.00211/) (2024)
- [People who frequently use ChatGPT for writing tasks are accurate and robust detectors of AI-generated text](https://mlsanity.com/paper/2501.15654/) (2025)
- [Evaluating the Evaluator: Measuring LLMs' Adherence to Task Evaluation Instructions](https://mlsanity.com/paper/2408.08781/) (2024)
- [LLM-based relevance assessment still can't replace human relevance assessment](https://mlsanity.com/paper/2412.17156/) (2024)
- [AI-Slop to AI-Polish? Aligning Language Models through Edit-Based Writing Rewards and Test-time Computation](https://mlsanity.com/paper/2504.07532/) (2025)
- [Are Large Language Models Reliable Judges? A Study on the Factuality Evaluation Capabilities of LLMs](https://mlsanity.com/paper/2311.00681/) (2023)
- [Can We Trust LLMs? Mitigate Overconfidence Bias in LLMs through Knowledge Transfer](https://mlsanity.com/paper/2405.16856/) (2024)
- [Self-Repetition in Abstractive Neural Summarizers](https://mlsanity.com/paper/2210.08145/) (2022)
- [QUDsim: Quantifying Discourse Similarities in LLM-Generated Text](https://mlsanity.com/paper/2504.09373/) (2025)
- [Flattery, Fluff, and Fog: Diagnosing and Mitigating Idiosyncratic Biases in Preference Models](https://mlsanity.com/paper/2506.05339/) (2025)
- [Do Automatic Factuality Metrics Measure Factuality? A Critical Evaluation](https://mlsanity.com/paper/2411.16638/) (2024)
- [Evaluating the Factuality of Zero-shot Summarizers Across Varied Domains](https://mlsanity.com/paper/2402.03509/) (2024)
- [Writing as a testbed for open ended agents](https://mlsanity.com/paper/2503.19711/) (2025)

## Built on by (most cited first, 6 of 6)

- [Death of the Novel(ty): Beyond n-Gram Novelty as a Metric for Textual Creativity](https://mlsanity.com/paper/2509.22641/) (2025)
- [AI generates well-liked but templatic empathic responses](https://mlsanity.com/paper/2604.08479/) (2026)
- [The Limits of Learning from Pictures and Text: Vision-Language Models and Embodied Scene Understanding](https://mlsanity.com/paper/2603.26589/) (2026)
- [Generative AI floods and dilutes the market for books](https://mlsanity.com/paper/2607.20349/) (2026)
- [Surface Reading LLMs: Synthetic Text and its Styles](https://mlsanity.com/paper/2510.22162/) (2025)
- [TRIM: Reducing AI-Generated CodeSlop via Agent Trajectory Minimization](https://mlsanity.com/paper/2607.18161/) (2026)

## Abstract (arXiv)

AI "slop" is an increasingly popular term used to describe low-quality AI-generated text, but there is currently no agreed upon definition of this term nor a means to measure its occurrence. In this work, we develop a taxonomy of "slop" through interviews with experts in NLP, writing, and philosophy, and propose a set of interpretable dimensions for its assessment in text. Through span-level annotation, we find that binary "slop" judgments are (somewhat) subjective, but such determinations nonetheless correlate with latent dimensions such as coherence and relevance. Our framework can be used to evaluate AI-generated text in both detection and binary preference tasks, potentially offering new insights into the linguistic and stylistic factors that contribute to quality judgments.