mlsanityData to 7 Oct 2026

Handling and Presenting Harmful Text in NLP Research

arXiv 2204.14256 ↗First posted 29 Apr 2022cs.CLcited by 43

Not on the map. Why?

Abstract (arXiv)

Text data can pose a risk of harm. However, the risks are not fully understood, and how to handle, present, and discuss harmful text in a safe way remains an unresolved issue in the NLP community. We provide an analytical framework categorising harms on three axes: (1) the harm type (e.g., misinformation, hate speech or racial stereotypes); (2) whether a harm is textitsought as a feature of the research design if explicitly studying harmful content (e.g., training a hate speech classifier), versus textitunsought if harmful content is encountered when working on unrelated problems (e.g., language generation or part-of-speech tagging); and (3) who it affects, from people (mis)represented in the data to those handling the data and those publishing on the data. We provide advice for practitioners, with concrete steps for mitigating harm in research and in publication. To assist implementation we introduce textscHarmCheck -- a documentation standard for handling and presenting harmful text in research.

Key facts. Cited by 43 papers in the map (data through 7 Oct 2026). Builds on 22 of its 73 references that are in the map. First posted on arXiv 29 Apr 2022.

Builds on

22 of 73 references.

References · 22

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  3. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  4. 2018Model Cards for Model Reportingcited by 1,068
  5. 2021Ethical and social risks of harm from Language Modelscited by 957
  6. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 951
  7. 2021Multimodal datasets: misogyny, pornography, and malignant stereotypescited by 326
  8. 2020Data and its (dis)contents: A survey of dataset development and use in machine learning researchcited by 259
  9. 2021Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detectioncited by 258
  10. 2021Quality at a Glance: An Audit of Web-Crawled Multilingual Datasetscited by 241
  11. 2020Large image datasets: A pyrrhic win for computer vision?cited by 237
  12. 2020HateCheck: Functional Tests for Hate Speech Detection Modelscited by 215
10 more
  1. 2021Bias Out-of-the-Box: An Empirical Analysis of Intersectional Occupational Biases in Popular Generative Language Modelscited by 141
  2. 2020Directions in Abusive Language Training Data: Garbage In, Garbage Outcited by 139
  3. 2019Lessons from Archives: Strategies for Collecting Sociocultural Data in Machine Learningcited by 116
  4. 2021Reduced, Reused and Recycled: The Life of a Dataset in Machine Learning Researchcited by 100
  5. 2020SemEval-2020 Task 11: Detection of Propaganda Techniques in News Articlescited by 79
  6. 2015Analysing How People Orient to and Spread Rumours in Social Media by Looking at Conversational Threadscited by 62
  7. 2021Hatemoji: A Test Suite and Adversarially-Generated Dataset for Benchmarking and Detecting Emoji-based Hatecited by 50
  8. 2021Just What do You Think You're Doing, Dave?' A Checklist for Responsible Data Use in NLPcited by 47
  9. 2020Against Scale: Provocations and Resistances to Scale Thinkingcited by 18
  10. 2021DataCLUE: A Benchmark Suite for Data-centric NLPcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 14; 2024: 12; 2025: 9; 2026: 5.

43 papers.

Sort:

By area:

  1. 2023Auditing large language models: a three-layered approachcited by 135
  2. 2022Evaluating Human-Language Model Interactioncited by 90
  3. 2022DataPerf: Benchmarks for Data-Centric AI Developmentcited by 89
  4. 2024Safeguarding Large Language Models: A Surveycited by 73
  5. 2023SemEval-2023 Task 10: Explainable Detection of Online Sexismcited by 48
  6. 2023You Are What You Annotate: Towards Better Models through Annotator Representationscited by 40
  7. 2023Assessing Language Model Deployment with Risk Cardscited by 30
  8. 2024"They are uncultured": Unveiling Covert Harms and Social Threats in LLM Generated Conversationscited by 24
  9. 2024NLP for Counterspeech against Hate: A Survey and How-To Guidecited by 21
  10. 2023Summon a Demon and Bind it: A Grounded Theory of LLM Red Teamingcited by 20
  11. 2024NLP Systems That Can't Tell Use from Mention Censor Counterspeech, but Teaching the Distinction Helpscited by 19
  12. 2023Cultural Compass: Predicting Transfer Learning Success in Offensive Language Detection with Cultural Featurescited by 18
31 more
  1. 2023Understanding Counterspeech for Online Harm Mitigationcited by 17
  2. 2024HateDay: Insights from a Global Hate Speech Dataset Representative of a Day on Twittercited by 12
  3. 2023DELPHI: Data for Evaluating LLMs' Performance in Handling Controversial Issuescited by 11
  4. 2025Reward Model Interpretability via Optimal and Pessimal Tokenscited by 10
  5. 2024Insights and Current Gaps in Open-Source LLM Vulnerability Scanners: A Comparative Analysiscited by 9
  6. 2023Seeing Seeds Beyond Weeds: Green Teaming Generative AI for Beneficial Usescited by 8
  7. 2023Adversarial Nibbler: A Data-Centric Challenge for Improving the Safety of Text-to-Image Modelscited by 7
  8. 2024Re-examining Sexism and Misogyny Classification with Annotator Attitudescited by 7
  9. 2024Generation with Dynamic Vocabularycited by 5
  10. 2025ABLEIST: Intersectional Disability Bias in LLM-Generated Hiring Scenarioscited by 4
  11. 2022Trigger Warnings: Bootstrapping a Violence Detector for FanFictioncited by 3
  12. 2023Toxic language detection: a systematic review of Arabic datasetscited by 3
  13. 2024Challenging Negative Gender Stereotypes: A Study on the Effectiveness of Automated Counter-Stereotypescited by 3
  14. 2024The Life Cycle of Large Language Models: A Review of Biases in Educationcited by 3
  15. 2025MythTriage: Scalable Detection of Opioid Use Disorder Myths on a Video-Sharing Platformcited by 3
  16. 2025IYKYK: Using language models to decode extremist cryptolectscited by 3
  17. 2026Persona-Grounded Safety Evaluation of AI Companions in Multi-Turn Conversationscited by 3
  18. 2023Exploiting Explainability to Design Adversarial Attacks and Evaluate Attack Resilience in Hate-Speech Detection Modelscited by 2
  19. 2025MindSET: Advancing Mental Health Benchmarking through Large-Scale Social Media Datacited by 2
  20. 2023Evaluating the Effectiveness of Natural Language Inference for Hate Speech Detection in Languages with Limited Labeled Datacited by 1
  21. 2024If there's a Trigger Warning, then where's the Trigger? Investigating Trigger Warnings at the Passage Levelcited by 1
  22. 2024When in Doubt, Cascade: Towards Building Efficient and Capable Guardrailscited by 1
  23. 2026SAFARI: A Community-Engaged Approach and Dataset of Stereotype Resources in the Sub-Saharan African Contextcited by 1
  24. 2026Going PLACES: Participatory Localized Red Teaming for Text-to-Image Safety in the Global Southcited by 1
  25. 2023DoDo Learning: DOmain-DemOgraphic Transfer in Language Models for Detecting Abuse Targeted at Public Figurescited by 0
  26. 2025Improving Hate Speech Classification with Cross-Taxonomy Dataset Integrationcited by 0
  27. 2025A Survey on Stereotype Detection in Natural Language Processingcited by 0
  28. 2025Before the Outrage: Challenges and Advances in Predicting Online Antisocial Behaviorcited by 0
  29. 2025Evaluating Machine Translation Datasets for Low-Web Data Languages: A Gendered Lenscited by 0
  30. 2026ToxiTrace: Gradient-Aligned Training for Explainable Chinese Toxicity Detectioncited by 0
  31. 2026SWARM: A Multilingual Human-Annotated Dataset for Russian Propaganda Detection in Search Engine Resultscited by 0