mlsanityData to 14 Oct 2026

Characteristics of Harmful Text: Towards Rigorous Benchmarking of Language Models

arXiv 2206.08325 ↗First posted 16 Jun 2022cs.CL cs.AI cs.CYcited by 49

Not on the map. Why?

Abstract (arXiv)

Large language models produce human-like text that drive a growing number of applications. However, recent literature and, increasingly, real world observations, have demonstrated that these models can generate language that is toxic, biased, untruthful or otherwise harmful. Though work to evaluate language model harms is under way, translating foresight about which harms may arise into rigorous benchmarks is not straightforward. To facilitate this translation, we outline six ways of characterizing harmful text which merit explicit consideration when designing new benchmarks. We then use these characteristics as a lens to identify trends and gaps in existing benchmarks. Finally, we apply them in a case study of the Perspective API, a toxicity classifier that is widely used in harm benchmarks. Our characteristics provide one piece of the bridge that translates between foresight and effective evaluation.

Key facts. Cited by 49 papers in the map (data through 14 Oct 2026). Builds on 58 of its 112 references that are in the map. First posted on arXiv 16 Jun 2022.

Builds on

58 of 112 references.

References · 58

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  4. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  5. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  6. 2019A Survey on Bias and Fairness in Machine Learningcited by 1,917
  7. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  8. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,312
  9. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,255
  10. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  11. 2018Model Cards for Model Reportingcited by 1,080
  12. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 1,006
38 more
  1. 2021Ethical and social risks of harm from Language Modelscited by 966
  2. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 956
  3. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 924
  4. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 799
  5. 2020Aligning AI With Shared Human Valuescited by 723
  6. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 710
  7. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 701
  8. 2018Gender Bias in Coreference Resolution: Evaluation and Debiasing Methodscited by 659
  9. 2019The Woman Worked as a Babysitter: On Biases in Language Generationcited by 541
  10. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 492
  11. 2018Gender Bias in Coreference Resolutioncited by 442
  12. 2022ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detectioncited by 422
  13. 2021BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generationcited by 412
  14. 2019Nuanced Metrics for Measuring Unintended Bias with Real Data for Text Classificationcited by 404
  15. 2020GeDi: Generative Discriminator Guided Sequence Generationcited by 378
  16. 2021Persistent Anti-Muslim Bias in Large Language Modelscited by 377
  17. 2021Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLPcited by 337
  18. 2021Towards Understanding and Mitigating Social Biases in Language Modelscited by 296
  19. 2022Challenges and Strategies in Cross-Cultural NLPcited by 218
  20. 2016Ex Machina: Personal Attacks Seen at Scalecited by 210
  21. 2021Societal Biases in Language Generation: Progress and Challengescited by 202
  22. 2016Demographic Dialectal Variation in Social Media: A Case Study of African-American Englishcited by 196
  23. 2021Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasetscited by 181
  24. 2021Challenges in Detoxifying Language Modelscited by 169
  25. 2021Alignment of Language Agentscited by 168
  26. 2019Reducing Sentiment Bias in Language Models via Counterfactual Evaluationcited by 148
  27. 2020Intrinsic Bias Metrics Do Not Correlate with Application Biascited by 148
  28. 2019Fairness-Aware Ranking in Search & Recommendation Systems with Application to LinkedIn Talent Searchcited by 138
  29. 2020Towards Accountability for Machine Learning Datasets: Practices from Software Engineering and Infrastructurecited by 138
  30. 2019Predictive Biases in Natural Language Processing Models: A Conceptual Framework and Overviewcited by 137
  31. 2020UnQovering Stereotyping Biases via Underspecified Questionscited by 126
  32. 2019Towards a Critical Race Methodology in Algorithmic Fairnesscited by 125
  33. 2020A Distributional Approach to Controlled Text Generationcited by 119
  34. 2021Detoxifying Language Models Risks Marginalizing Minority Voicescited by 107
  35. 2021Re-imagining Algorithmic Fairness in India and Beyondcited by 104
  36. 2020Decolonial AI: Decolonial Theory as Sociotechnical Foresight in Artificial Intelligencecited by 101
  37. 2020Investigating African-American Vernacular English in Transformer-Based Text Generationcited by 53
  38. 2020Toxic Language Detection in Social Media for Brazilian Portuguese: New Dataset and Multilingual Analysiscited by 45

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 3; 2023: 16; 2024: 16; 2025: 9; 2026: 5.

49 papers.

Sort:

By area:

  1. 2023PaLM 2 Technical Reportcited by 1,162
  2. 2022Solving math word problems with process- and outcome-based feedbackcited by 901
  3. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 860
  4. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 656
  5. 2022Improving alignment of dialogue agents via targeted human judgementscited by 531
  6. 2023Towards Measuring the Representation of Subjective Global Opinions in Language Modelscited by 437
  7. 2024When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMscited by 240
  8. 2023Can LLM-Generated Misinformation Be Detected?cited by 177
  9. 2023The Capacity for Moral Self-Correction in Large Language Modelscited by 173
  10. 2023Conformal Language Modelingcited by 170
  11. 2023Auditing large language models: a three-layered approachcited by 137
  12. 2024Token-level Direct Preference Optimizationcited by 135
37 more
  1. 2023Evaluating the Social Impact of Generative AI Systems in Systems and Societycited by 92
  2. 2022Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Surveycited by 80
  3. 2024CoSafe: Evaluating Large Language Model Safety in Multi-Turn Dialogue Coreferencecited by 51
  4. 2025Multilingual != Multicultural: Evaluating Gaps Between Multilingual Capabilities and Cultural Alignment in LLMscited by 39
  5. 2024RTP-LX: Can LLMs Evaluate Toxicity in Multilingual Scenarios?cited by 36
  6. 2024A Robot Walks into a Bar: Can Language Models Serve as Creativity Support Tools for Comedy? An Evaluation of LLMs' Humour Alignment with Comedianscited by 29
  7. 2023Unveiling Safety Vulnerabilities of Large Language Modelscited by 27
  8. 2024The Impossibility of Fair LLMscited by 22
  9. 2023An Evaluation on Large Language Model Outputs: Discourse and Memorizationcited by 20
  10. 2023Appraising the Potential Uses and Harms of LLMs for Medical Systematic Reviewscited by 18
  11. 2025Can LLMs Correct Themselves? A Benchmark of Self-Correction in LLMscited by 14
  12. 2024Generative Echo Chamber? Effects of LLM-Powered Search Systems on Diverse Information Seekingcited by 13
  13. 2024STAR: SocioTechnical Approach to Red Teaming Language Modelscited by 13
  14. 2026Shaping capabilities with token-level data filteringcited by 13
  15. 2023Towards the Scalable Evaluation of Cooperativeness in Language Modelscited by 12
  16. 2024"Cold, Calculated, and Condescending": How AI Identifies and Explains Ableism Compared to Disabled Peoplecited by 10
  17. 2025SAGE: A Generic Framework for LLM Safety Evaluationcited by 8
  18. 2024Imposter.AI: Adversarial Attacks with Hidden Intentions towards Aligned Large Language Modelscited by 7
  19. 2024Reward-RAG: Enhancing RAG with Reward Driven Supervisioncited by 7
  20. 2025Taxonomizing Representational Harms using Speech Act Theorycited by 7
  21. 2025Mitigating Biases in Language Models via Bias Unlearningcited by 6
  22. 2023Who's Thinking? A Push for Human-Centered Evaluation of LLMs using the XAI Playbookcited by 5
  23. 2024Laissez-Faire Harms: Algorithmic Biases in Generative Language Modelscited by 5
  24. 2025Understanding Gender Bias in AI-Generated Product Descriptionscited by 3
  25. 2025Being Kind Isn't Always Being Safe: Diagnosing Affective Hallucination in LLMscited by 3
  26. 2026Evaluating Human-AI Safety: A Framework for Measuring Harmful Capability Upliftcited by 3
  27. 2024GeniL: A Multilingual Dataset on Generalizing Languagecited by 2
  28. 2025SEALGuard: Safeguarding the Multilingual Conversations in Southeast Asian Languages for LLM Software Systemscited by 2
  29. 2025Falcon: A Cross-Modal Evaluation Dataset for Comprehensive Safety Perceptioncited by 2
  30. 2023Operationalising the Definition of General Purpose AI Systems: Assessing Four Approachescited by 1
  31. 2024Soft-Label Integration for Robust Toxicity Classificationcited by 1
  32. 2024Ethics Whitepaper: Whitepaper on Ethical Research into Large Language Modelscited by 1
  33. 2026Food Noise & False Safety: A Systematic Evaluation of How LLMs Fail to Adapt to Eating Disorder Queries with Clinician Feedbackcited by 1
  34. 2023Apolitical Intelligence? Auditing Delphi's responses on controversial political issues in the UScited by 0
  35. 2024Evaluating the Propensity of Generative AI for Producing Harmful Disinformation During the 2024 US Election Cyclecited by 0
  36. 2026AERIC: Anticipatory Hidden-State Monitoring for Implicit Harmful Dialoguecited by 0
  37. 2026Grounding Text Embeddings in Stakeholder Associationscited by 0