mlsanityData to 7 Oct 2026

Bye Bye Perspective API: Lessons for Building and Governing Measurement Infrastructure

arXiv 2604.25580 ↗First posted 28 Apr 2026cs.CLcited by 1

Not on the map. Why?

Abstract (arXiv)

Perspective API closes at the end of 2026, removing the de facto standard for toxicity measurement and exposing researchers' dependence on a tool they did not control. Drawing on this case, we argue that a research field must build and govern its own measurement infrastructure rather than borrow it. Surveying 241 papers that use or study Perspective, we show what depending on it cost the research community: claims reaching past what the tool could support, results that shifted when its model was silently retrained, and disparities researchers could measure but not explain. These failures were amplified throughout the LLM lifecycle, where Perspective supplied the labels, filtered the corpora, and graded the systems trained on each, rewarding errors rather than catching them. To keep the instrument open to study after shutdown, we release Perspective scores for 5.9 million text snippets from 77 datasets. We further specify ten requirements for measurement infrastructure a field owns, and argue that what blocks such infrastructure is not technical capability but the value the field places on infrastructure work.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 75 of its 156 references that are in the map. First posted on arXiv 28 Apr 2026.

Builds on

75 of 156 references.

References · 75

  1. 2017On Calibration of Modern Neural Networkscited by 4,860
  2. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  3. 2018Hierarchical Neural Story Generationcited by 1,182
  4. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 951
  5. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 793
  6. 2017Automated Hate Speech Detection and the Problem of Offensive Languagecited by 724
  7. 2021Dynabench: Rethinking Benchmarking in NLPcited by 447
  8. 2021DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Expertscited by 434
  9. 2020HateXplain: A Benchmark Dataset for Explainable Hate Speech Detectioncited by 403
  10. 2019Nuanced Metrics for Measuring Unintended Bias with Real Data for Text Classificationcited by 396
  11. 2019Measurement and Fairnesscited by 299
  12. 2021Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detectioncited by 258
38 more
  1. 2023Pretraining Language Models with Human Preferencescited by 245
  2. 2023ToxicChat: Unveiling Hidden Challenges of Toxicity Detection in Real-World User-AI Conversationcited by 224
  3. 2020HateCheck: Functional Tests for Hate Speech Detection Modelscited by 215
  4. 2016Ex Machina: Personal Attacks Seen at Scalecited by 208
  5. 2022A New Generation of Perspective API: Efficient Multilingual Character-level Transformerscited by 181
  6. 2023A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicitycited by 180
  7. 2020Directions in Abusive Language Training Data: Garbage In, Garbage Outcited by 139
  8. 2019CONAN -- COunter NArratives through Nichesourcing: a Multilingual Dataset of Responses to Fight Online Hate Speechcited by 109
  9. 2017Measuring the Reliability of Hate Speech Annotations: The Case of the European Refugee Crisiscited by 95
  10. 2018Conversations Gone Awry: Detecting Early Signs of Conversational Failurecited by 73
  11. 2021Agreeing to Disagree: Annotating Offensive Language Datasets with Annotators' Disagreementcited by 71
  12. 2023Word Embeddings Are Steers for Language Modelscited by 70
  13. 2020ETHOS: an Online Hate Speech Detection Datasetcited by 66
  14. 2022Mix and Match: Learning-free Controllable Text Generation using Energy Language Modelscited by 66
  15. 2025Position: Evaluating Generative AI Systems Is a Social Science Measurement Challengecited by 61
  16. 2022Handling Bias in Toxic Speech Detection: A Surveycited by 60
  17. 2021Hatemoji: A Test Suite and Adversarially-Generated Dataset for Benchmarking and Detecting Emoji-based Hatecited by 50
  18. 2023SemEval-2023 Task 10: Explainable Detection of Online Sexismcited by 48
  19. 2020Toxic Language Detection in Social Media for Brazilian Portuguese: New Dataset and Multilingual Analysiscited by 45
  20. 2023On the Challenges of Using Black-Box APIs for Toxicity Evaluation in Researchcited by 44
  21. 2022Large-Scale Hate Speech Detection with Cross-Domain Transfercited by 39
  22. 2021Mitigating harm in language models with conditional-likelihood filtrationcited by 38
  23. 2022Hate Speech and Counter Speech Detection: Conversational Context Does Mattercited by 32
  24. 2021Detecting Cross-Geographic Biases in Toxicity Modeling on Social Mediacited by 31
  25. 2023From Dogwhistles to Bullhorns: Unveiling Coded Rhetoric with Language Modelscited by 26
  26. 2022How Hate Speech Varies by Target Identity: A Computational Analysiscited by 23
  27. 2024From Languages to Geographies: Towards Evaluating Cultural Bias in Hate Speech Datasetscited by 23
  28. 2021The Structure of Toxic Conversations on Twittercited by 22
  29. 2022Listening to Affected Communities to Define Extreme Speech: Dataset and Experimentscited by 21
  30. 2023Toxic Bias: Perspective API Misreads German as More Toxiccited by 19
  31. 2024NLP Systems That Can't Tell Use from Mention Censor Counterspeech, but Teaching the Distinction Helpscited by 19
  32. 2024Controlled Text Generation for Large Language Model with Dynamic Attribute Graphscited by 16
  33. 2022HateCheckHIn: Evaluating Hindi Hate Speech Detection Modelscited by 15
  34. 2024CantTalkAboutThis: Aligning Language Models to Stay on Topic in Dialoguescited by 15
  35. 2023YouNICon: YouTube's CommuNIty of Conspiracy Videoscited by 13
  36. 2024Improving Adversarial Data Collection by Supporting Annotators: Lessons from GAHD, a German Hate Speech Datasetcited by 13
  37. 2024HateDay: Insights from a Global Hate Speech Dataset Representative of a Day on Twittercited by 12
  38. 2024Multilingual and Explainable Text Detoxification with Parallel Corporacited by 12

50 most cited of 75 All 75 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.