# Critical Perspectives: A Benchmark Revealing Pitfalls in PerspectiveAPI

arXiv: https://arxiv.org/abs/2301.01874 · HTML: https://mlsanity.com/paper/2301.01874/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 36 references that are in the map.
- First posted on arXiv 5 Jan 2023.

## Builds on (most cited first, 13 of 13)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classification](https://mlsanity.com/paper/2010.12421/) (2020)
- [Mitigating Gender Bias in Natural Language Processing: Literature Review](https://mlsanity.com/paper/1906.08976/) (2019)
- [Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detection](https://mlsanity.com/paper/2012.15761/) (2020)
- [HateCheck: Functional Tests for Hate Speech Detection Models](https://mlsanity.com/paper/2012.15606/) (2020)
- [Ex Machina: Personal Attacks Seen at Scale](https://mlsanity.com/paper/1610.08914/) (2016)
- [A New Generation of Perspective API: Efficient Multilingual Character-level Transformers](https://mlsanity.com/paper/2202.11176/) (2022)
- [Deceiving Google's Perspective API Built for Detecting Toxic Comments](https://mlsanity.com/paper/1702.08138/) (2017)
- [All You Need is "Love": Evading Hate-speech Detection](https://mlsanity.com/paper/1808.09115/) (2018)
- [Anyone Can Become a Troll: Causes of Trolling Behavior in Online Discussions](https://mlsanity.com/paper/1702.01119/) (2017)
- [Towards Robust Toxic Content Classification](https://mlsanity.com/paper/1912.06872/) (2019)
- [Ground-Truth, Whose Truth? -- Examining the Challenges with Annotating Toxic Text Datasets](https://mlsanity.com/paper/2112.03529/) (2021)

## Built on by (most cited first, 6 of 6)

- [Safeguarding Large Language Models: A Survey](https://mlsanity.com/paper/2406.02622/) (2024)
- [Guardians and Offenders: A Survey on Harmful Content Generation and Safety Mitigation of LLM](https://mlsanity.com/paper/2508.05775/) (2025)
- [Testing the Limits of Jailbreaking Defenses with the Purple Problem](https://mlsanity.com/paper/2403.14725/) (2024)
- [The Monetisation of Toxicity: Analysing YouTube Content Creators and Controversy-Driven Engagement](https://mlsanity.com/paper/2408.00534/) (2024)
- [Something Just Like TRuST : Toxicity Recognition of Span and Target](https://mlsanity.com/paper/2506.02326/) (2025)
- [Characterization of Political Polarized Users Attacked by Language Toxicity on Twitter](https://mlsanity.com/paper/2407.12471/) (2024)

## Abstract (arXiv)

Detecting "toxic" language in internet content is a pressing social and technical challenge. In this work, we focus on PERSPECTIVE from Jigsaw, a state-of-the-art tool that promises to score the "toxicity" of text, with a recent model update that claims impressive results (Lees et al., 2022). We seek to challenge certain normative claims about toxic language by proposing a new benchmark, Selected Adversarial SemanticS, or SASS. We evaluate PERSPECTIVE on SASS, and compare to low-effort alternatives, like zero-shot and few-shot GPT-3 prompt models, in binary classification settings. We find that PERSPECTIVE exhibits troubling shortcomings across a number of our toxicity categories. SASS provides a new tool for evaluating performance on previously undetected toxic language that avoids common normative pitfalls. Our work leads us to emphasize the importance of questioning assumptions made by tools already in deployment for toxicity detection in order to anticipate and prevent disparate harms.