# To Err is AI : A Case Study Informing LLM Flaw Reporting Practices

arXiv: https://arxiv.org/abs/2410.12104 · HTML: https://mlsanity.com/paper/2410.12104/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 39 references that are in the map.
- First posted on arXiv 15 Oct 2024.

## Builds on (most cited first, 22 of 22)

- [Jailbreaking Black Box Large Language Models in Twenty Queries](https://mlsanity.com/paper/2310.08419/) (2023)
- [Datasheets for Datasets](https://mlsanity.com/paper/1803.09010/) (2018)
- [Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!](https://mlsanity.com/paper/2310.03693/) (2023)
- [Model Cards for Model Reporting](https://mlsanity.com/paper/1810.03993/) (2018)
- [Universal Adversarial Triggers for Attacking and Analyzing NLP](https://mlsanity.com/paper/1908.07125/) (2019)
- [WildChat: 1M ChatGPT Interaction Logs in the Wild](https://mlsanity.com/paper/2405.01470/) (2024)
- [OLMo: Accelerating the Science of Language Models](https://mlsanity.com/paper/2402.00838/) (2024)
- [GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts](https://mlsanity.com/paper/2309.10253/) (2023)
- [Scalable Extraction of Training Data from (Production) Language Models](https://mlsanity.com/paper/2311.17035/) (2023)
- [Low-Resource Languages Jailbreak GPT-4](https://mlsanity.com/paper/2310.02446/) (2023)
- [MART: Improving LLM Safety with Multi-round Automatic Red-Teaming](https://mlsanity.com/paper/2311.07689/) (2023)
- [Understanding Catastrophic Forgetting in Language Models via Implicit Inference](https://mlsanity.com/paper/2309.10105/) (2023)
- [Black-Box Access is Insufficient for Rigorous AI Audits](https://mlsanity.com/paper/2401.14446/) (2024)
- [Preventing Repeated Real World AI Failures by Cataloging Incidents: The AI Incident Database](https://mlsanity.com/paper/2011.08512/) (2020)
- [On the Societal Impact of Open Foundation Models](https://mlsanity.com/paper/2403.07918/) (2024)
- [Opening up ChatGPT: Tracking openness, transparency, and accountability in instruction-tuned text generators](https://mlsanity.com/paper/2307.05532/) (2023)
- [A Safe Harbor for AI Evaluation and Red Teaming](https://mlsanity.com/paper/2403.04893/) (2024)
- [Consent in Crisis: The Rapid Decline of the AI Data Commons](https://mlsanity.com/paper/2407.14933/) (2024)
- [AI auditing: The Broken Bus on the Road to AI Accountability](https://mlsanity.com/paper/2401.14462/) (2024)
- [Can large language models democratize access to dual-use biotechnology?](https://mlsanity.com/paper/2306.03809/) (2023)
- [ChatGPT as an Attack Tool: Stealthy Textual Backdoor Attack via Blackbox Generative Model Trigger](https://mlsanity.com/paper/2304.14475/) (2023)
- [Coordinated Flaw Disclosure for AI: Beyond Security Vulnerabilities](https://mlsanity.com/paper/2402.07039/) (2024)

## Built on by (most cited first, 5 of 5)

- [In-House Evaluation Is Not Enough: Towards Robust Third-Party Flaw Disclosure for General-Purpose AI](https://mlsanity.com/paper/2503.16861/) (2025)
- [Benchmarking LLM-Assisted Blue Teaming via Standardized Threat Hunting](https://mlsanity.com/paper/2509.23571/) (2025)
- [Malicious and Unintentional Disclosure Risks in Large Language Models for Code Generation](https://mlsanity.com/paper/2503.22760/) (2025)
- [FLARE-AI: Flaw Reporting for AI](https://mlsanity.com/paper/2606.31567/) (2026)
- [Mapping the RAG Landscape: A Four Axis Taxonomy of Efficiency, Defense, Interactivity, and Reasoning](https://mlsanity.com/paper/2610.01936/) (2026)

## Abstract (arXiv)

In August of 2024, 495 hackers generated evaluations in an open-ended bug bounty targeting the Open Language Model (OLMo) from The Allen Institute for AI. A vendor panel staffed by representatives of OLMo's safety program adjudicated changes to OLMo's documentation and awarded cash bounties to participants who successfully demonstrated a need for public disclosure clarifying the intent, capacities, and hazards of model deployment. This paper presents a collection of lessons learned, illustrative of flaw reporting best practices intended to reduce the likelihood of incidents and produce safer large language models (LLMs). These include best practices for safety reporting processes, their artifacts, and safety program staffing.