# RealHarm: A Collection of Real-World Language Model Application Failures

arXiv: https://arxiv.org/abs/2504.10277 · HTML: https://mlsanity.com/paper/2504.10277/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 29 references that are in the map.
- First posted on arXiv 14 Apr 2025.

## Builds on (most cited first, 11 of 11)

- [HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal](https://mlsanity.com/paper/2402.04249/) (2024)
- [FEVER: a large-scale dataset for Fact Extraction and VERification](https://mlsanity.com/paper/1803.05355/) (2018)
- [A Holistic Approach to Undesired Content Detection in the Real World](https://mlsanity.com/paper/2208.03274/) (2022)
- [ToxicChat: Unveiling Hidden Challenges of Toxicity Detection in Real-World User-AI Conversation](https://mlsanity.com/paper/2310.17389/) (2023)
- [A Survey of Privacy Attacks in Machine Learning](https://mlsanity.com/paper/2007.07646/) (2020)
- [AEGIS: Online Adaptive AI Content Safety Moderation with Ensemble of LLM Experts](https://mlsanity.com/paper/2404.05993/) (2024)
- [Preventing Repeated Real World AI Failures by Cataloging Incidents: The AI Incident Database](https://mlsanity.com/paper/2011.08512/) (2020)
- [Introducing v0.5 of the AI Safety Benchmark from MLCommons](https://mlsanity.com/paper/2404.12241/) (2024)
- [Preserving Privacy in Large Language Models: A Survey on Current Threats and Solutions](https://mlsanity.com/paper/2408.05212/) (2024)
- [BELLS: A Framework Towards Future Proof Benchmarks for the Evaluation of LLM Safeguards](https://mlsanity.com/paper/2406.01364/) (2024)
- [A taxonomic system for failure cause analysis of open source AI incidents](https://mlsanity.com/paper/2211.07280/) (2022)

## Built on by (most cited first, 8 of 8)

- [XGUARD: A Graded Benchmark for Evaluating Safety Failures of Large Language Models on Extremist Content](https://mlsanity.com/paper/2506.00973/) (2025)
- [A Red Teaming Roadmap Towards System-Level Safety](https://mlsanity.com/paper/2506.05376/) (2025)
- [Teaching People LLM's Errors and Getting it Right](https://mlsanity.com/paper/2512.21422/) (2025)
- [To Build or Not to Build? Factors that Lead to Non-Development or Abandonment of AI Systems](https://mlsanity.com/paper/2604.28053/) (2026)
- [SciHazard: A Benchmark for Measuring Scientific Safety Risks with Decomposed Harm Scoring](https://mlsanity.com/paper/2607.18665/) (2026)
- [Specification and Detection of LLM Code Smells](https://mlsanity.com/paper/2512.18020/) (2025)
- [StereoTales: A Multilingual Framework for Open-Ended Stereotype Discovery in LLMs](https://mlsanity.com/paper/2605.10442/) (2026)
- [Mining Workflow Graphs for Black-Box Boundary Testing of Conversational LLM Agents](https://mlsanity.com/paper/2607.06873/) (2026)

## Abstract (arXiv)

Language model deployments in consumer-facing applications introduce numerous risks. While existing research on harms and hazards of such applications follows top-down approaches derived from regulatory frameworks and theoretical analyses, empirical evidence of real-world failure modes remains underexplored. In this work, we introduce RealHarm, a dataset of annotated problematic interactions with AI agents built from a systematic review of publicly reported incidents. Analyzing harms, causes, and hazards specifically from the deployer's perspective, we find that reputational damage constitutes the predominant organizational harm, while misinformation emerges as the most common hazard category. We empirically evaluate state-of-the-art guardrails and content moderation systems to probe whether such systems would have prevented the incidents, revealing a significant gap in the protection of AI applications.