mlsanityData to 14 Oct 2026

Can LLMs Rank the Harmfulness of Smaller LLMs? We are Not There Yet

arXiv 2502.05291 ↗First posted 7 Feb 2025cs.CLcited by 0

On the map: LLM safety and security → Safety alignment, refusal and guardrails

Abstract (arXiv)

Large language models (LLMs) have become ubiquitous, thus it is important to understand their risks and limitations. Smaller LLMs can be deployed where compute resources are constrained, such as edge devices, but with different propensity to generate harmful output. Mitigation of LLM harm typically depends on annotating the harmfulness of LLM output, which is expensive to collect from humans. This work studies two questions: How do smaller LLMs rank regarding generation of harmful content? How well can larger LLMs annotate harmfulness? We prompt three small LLMs to elicit harmful content of various types, such as discriminatory language, offensive content, privacy invasion, or negative influence, and collect human rankings of their outputs. Then, we evaluate three state-of-the-art large LLMs on their ability to annotate the harmfulness of these responses. We find that the smaller models differ with respect to harmfulness. We also find that large LLMs show low to moderate agreement with humans. These findings underline the need for further work on harm mitigation in LLMs.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security. Builds on 36 of its 54 references that are in the map. First posted on arXiv 7 Feb 2025.

Builds on

36 of 54 references.

References · 36

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  5. 2020Extracting Training Data from Large Language Modelscited by 2,189
  6. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  7. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,255
  8. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  9. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 870
  10. 2017Automated Hate Speech Detection and the Problem of Offensive Languagecited by 736
  11. 2019Green AIcited by 627
  12. 2024AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agentscited by 437
24 more
  1. 2023Toxicity in ChatGPT: Analyzing Persona-assigned Language Modelscited by 360
  2. 2024WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wildcited by 218
  3. 2020HateBERT: Retraining BERT for Abusive Language Detection in Englishcited by 207
  4. 2022What Does it Mean for a Language Model to Preserve Privacy?cited by 195
  5. 2022A New Generation of Perspective API: Efficient Multilingual Character-level Transformerscited by 185
  6. 2023AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotatorscited by 172
  7. 2024Secrets of RLHF in Large Language Models Part II: Reward Modelingcited by 171
  8. 2024On-Device Language Models: A Comprehensive Reviewcited by 104
  9. 2023Can ChatGPT Reproduce Human-Generated Labels? A Study of Social Computing Taskscited by 89
  10. 2024Detoxifying Large Language Models via Knowledge Editingcited by 87
  11. 2021On Measures of Biases and Harms in NLPcited by 81
  12. 2022Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Surveycited by 80
  13. 2023LLMaAA: Making Large Language Models as Active Annotatorscited by 79
  14. 2021A Survey of Race, Racism, and Anti-Racism in NLPcited by 75
  15. 2024MEGAnno+: A Human-LLM Collaborative Annotation Systemcited by 47
  16. 2024If in a Crowdsourced Data Annotation Pipeline, a GPT-4cited by 38
  17. 2024The Effectiveness of LLMs as Annotators: A Comparative Overview and Empirical Analysis of Direct Representationcited by 29
  18. 2023Sociodemographic Bias in Language Models: A Survey and Forward Pathcited by 28
  19. 2022Gendered Mental Health Stigma in Masked Language Modelscited by 18
  20. 2024Parameter-Efficient Detoxification with Contrastive Decodingcited by 18
  21. 2023CALM : A Multi-task Benchmark for Comprehensive Assessment of Language Model Biascited by 14
  22. 2024DetoxLLM: A Framework for Detoxification with Explanationscited by 10
  23. 2024Large Language Models can be Strong Self-Detoxifierscited by 7
  24. 2024Hey GPT, Can You be More Racist? Analysis from Crowdsourced Attempts to Elicit Biased Content from Generative AIcited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025HarmMetric Eval: Benchmarking Metrics and Judges for LLM Harmfulness Assessment
  2. 2026HarmProfile: Characterizing Harmful Distributions in Frontier LLMs
  3. 2025A Survey on Responsible LLMs: Inherent Risk, Malicious Use, and Mitigation Strategy
  4. 2025Towards Safer Pretraining: Analyzing and Filtering Harmful Content in Webscale datasets for Responsible LLMs
  5. 2026Understanding LLM Behavior When Encountering User-Supplied Harmful Content in Harmless Tasks
  6. 2025Guardians and Offenders: A Survey on Harmful Content Generation and Safety Mitigation of LLM
  7. 2026No One Model Catches Every Harm: Benchmarking Content Moderation Across Safety Scenarios
  8. 2024SafetyPrompts: a Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safety