mlsanityData to 7 Oct 2026

Watching the AI Watchdogs: A Fairness and Robustness Analysis of AI Safety Moderation Classifiers

arXiv 2501.13302 ↗First posted 23 Jan 2025cs.CL cs.AIcited by 5

On the map: LLM safety and security → Safety alignment, refusal and guardrails

Abstract (arXiv)

AI Safety Moderation (ASM) classifiers are designed to moderate content on social media platforms and to serve as guardrails that prevent Large Language Models (LLMs) from being fine-tuned on unsafe inputs. Owing to their potential for disparate impact, it is crucial to ensure that these classifiers: (1) do not unfairly classify content belonging to users from minority groups as unsafe compared to those from majority groups and (2) that their behavior remains robust and consistent across similar inputs. In this work, we thus examine the fairness and robustness of four widely-used, closed-source ASM classifiers: OpenAI Moderation API, Perspective API, Google Cloud Natural Language (GCNL) API, and Clarifai API. We assess fairness using metrics such as demographic parity and conditional statistical parity, comparing their performance against ASM models and a fair-only baseline. Additionally, we analyze robustness by testing the classifiers' sensitivity to small and natural input perturbations. Our findings reveal potential fairness and robustness gaps, highlighting the need to mitigate these issues in future versions of these models.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security. Builds on 22 of its 46 references that are in the map. First posted on arXiv 23 Jan 2025.

Builds on

22 of 46 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!5 Oct

Other references · 21

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  5. 2019A Survey on Bias and Fairness in Machine Learningcited by 1,911
  6. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  7. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  8. 2017Counterfactual Fairnesscited by 802
  9. 2015Improving Neural Machine Translation Models with Monolingual Datacited by 784
  10. 2022BERTopic: Neural topic modeling with a class-based TF-IDF procedurecited by 779
  11. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 622
  12. 2019The Woman Worked as a Babysitter: On Biases in Language Generationcited by 540
9 more
  1. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 533
  2. 2023Towards Measuring the Representation of Subjective Global Opinions in Language Modelscited by 424
  3. 2017Algorithmic decision making and the cost of fairnesscited by 402
  4. 2023A Comprehensive Capability Analysis of GPT-3 and GPT-3.5 Series Modelscited by 244
  5. 2024Measuring Political Bias in Large Language Models: What Is Said and How It Is Saidcited by 89
  6. 2024Revisiting Zero-Shot Abstractive Summarization in the Era of Large Language Models from the Perspective of Position Biascited by 28
  7. 2023Toxic Bias: Perspective API Misreads German as More Toxiccited by 19
  8. 2022Robust Fair Clustering: A Novel Fairness Attack and Defense Frameworkcited by 13
  9. 2024Incentivizing News Consumption on Social Media Platforms Using Large Language Models and Realistic Bot Accountscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 2.

5 papers.

  1. 2025Less Diverse, Less Safe: The Indirect But Pervasive Risk of Test-Time Scaling in Large Language Modelscited by 2
  2. 2025"Whose Side Are You On?" Estimating Ideology of Political and News Content Using Large Language Models and Few-shot Demonstration Selectioncited by 1
  3. 2025Bridging the Culture Gap: A Framework for LLM-Driven Socio-Cultural Localization of Math Word Problems in Low-Resource Languagescited by 1
  4. 2026To Lie or Not to Lie? Investigating The Biased Spread of Global Lies by LLMscited by 0
  5. 2026Efficient LLM Adversarial Training via Low-Rank Defense and Circuit-Guided Surrogatescited by 0