mlsanityData to 7 Oct 2026

A Safe Harbor for AI Evaluation and Red Teaming

arXiv 2403.04893 ↗First posted 7 Mar 2024cs.AIcited by 46

On the map: LLM safety and security → Prompt injection and agent security

Abstract (arXiv)

Independent evaluation and red teaming are critical for identifying the risks posed by generative AI systems. However, the terms of service and enforcement strategies used by prominent AI companies to deter model misuse have disincentives on good faith safety evaluations. This causes some researchers to fear that conducting such research or releasing their findings will result in account suspensions or legal reprisal. Although some companies offer researcher access programs, they are an inadequate substitute for independent research access, as they have limited community representation, receive inadequate funding, and lack independence from corporate incentives. We propose that major AI developers commit to providing a legal and technical safe harbor, indemnifying public interest safety research and protecting it from the threat of account suspensions or legal reprisal. These proposals emerged from our collective experience conducting safety, privacy, and trustworthiness research on generative AI systems, where norms and incentives could be better aligned with public interests, without exacerbating model misuse. We believe these commitments are a necessary step towards more inclusive and unimpeded community efforts to tackle the risks of generative AI.

Key facts. Cited by 46 papers in the map (data through 7 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 48 of its 131 references that are in the map. First posted on arXiv 7 Mar 2024.

Builds on

48 of 131 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023MART: Improving LLM Safety with Multi-round Automatic Red-Teaming13 Nov
  2. 2023Goal-Oriented Prompt Attack and Safety Evaluation for LLMs21 Sep
  3. 2023Multi-step Jailbreaking Privacy Attacks on ChatGPT11 Apr

Other references · 45

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  3. 2020Extracting Training Data from Large Language Modelscited by 2,144
  4. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  5. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  6. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  7. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  8. 2023Towards Understanding Sycophancy in Language Modelscited by 988
  9. 2023Whose Opinions Do Language Models Reflect?cited by 759
  10. 2023Extracting Training Data from Diffusion Modelscited by 723
  11. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 533
  12. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 520
33 more
  1. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 457
  2. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 439
  3. 2023Scalable Extraction of Training Data from (Production) Language Modelscited by 415
  4. 2023Detecting Pretraining Data from Large Language Modelscited by 375
  5. 2023Toxicity in ChatGPT: Analyzing Persona-assigned Language Modelscited by 356
  6. 2023How Language Model Hallucinations Can Snowballcited by 310
  7. 2023Low-Resource Languages Jailbreak GPT-4cited by 290
  8. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Modelscited by 284
  9. 2022Red-Teaming the Stable Diffusion Safety Filtercited by 257
  10. 2023An Overview of Catastrophic AI Riskscited by 214
  11. 2023Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulationcited by 202
  12. 2023Unsafe Diffusion: On the Generation of Unsafe Images and Hateful Memes From Text-To-Image Modelscited by 171
  13. 2023Removing RLHF Protections in GPT-4 via Fine-Tuningcited by 165
  14. 2023Sociotechnical Safety Evaluation of Generative AI Systemscited by 154
  15. 2023Understanding Catastrophic Forgetting in Language Models via Implicit Inferencecited by 138
  16. 2023Emergent autonomous scientific research capabilities of large language modelscited by 127
  17. 2024Black-Box Access is Insufficient for Rigorous AI Auditscited by 123
  18. 2023Foundation Models and Fair Usecited by 112
  19. 2024LLM Agents can Autonomously Hack Websitescited by 108
  20. 2023The Earth is Flat because...: Investigating LLMs' Belief towards Misinformation via Persuasive Conversationcited by 107
  21. 2023The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution in AIcited by 73
  22. 2022Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governancecited by 67
  23. 2024On the Societal Impact of Open Foundation Modelscited by 64
  24. 2023Black Box Adversarial Prompting for Foundation Modelscited by 62
  25. 2023Who Audits the Auditors? Recommendations from a field scan of the algorithmic auditing ecosystemcited by 49
  26. 2024AI auditing: The Broken Bus on the Road to AI Accountabilitycited by 42
  27. 2023Can large language models democratize access to dual-use biotechnology?cited by 33
  28. 2023Artificial Influence: An Analysis Of AI-Driven Persuasioncited by 31
  29. 2023Privacy Implications of Retrieval-Based Language Modelscited by 31
  30. 2023ChatGPT as an Attack Tool: Stealthy Textual Backdoor Attack via Blackbox Generative Model Triggercited by 29
  31. 2024Coordinated Flaw Disclosure for AI: Beyond Security Vulnerabilitiescited by 8
  32. 2023Adversarial Nibbler: A Data-Centric Challenge for Improving the Safety of Text-to-Image Modelscited by 7
  33. 2023Bug Hunters' Perspectives on the Challenges and Benefits of the Bug Bounty Ecosystemcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 19; 2025: 21; 2026: 6.

46 papers.

Sort:

By area:

  1. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 494
  2. 2024ALERT: A Comprehensive Benchmark for Assessing Large Language Models' Safety through Red Teamingcited by 96
  3. 2024The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Modelscited by 89
  4. 2024Preference Tuning For Toxicity Mitigation Generalizes Across Languagescited by 31
  5. 2025Adversarial Attacks on Robotic Vision Language Action Modelscited by 31
  6. 2024Exploring Safety-Utility Trade-Offs in Personalized Language Modelscited by 26
  7. 2025RAG LLMs are Not Safer: A Safety Analysis of Retrieval-Augmented Generation for Large Language Modelscited by 23
  8. 2024Can Editing LLMs Inject Harm?cited by 22
  9. 2024The 2024 Foundation Model Transparency Indexcited by 18
  10. 2024Lies, Damned Lies, and Distributional Language Statistics: Persuasion and Deception with Large Language Modelscited by 14
  11. 2024The Responsible Foundation Model Development Cheatsheet: A Review of Tools & Resourcescited by 11
  12. 2024Responsible Reporting for Frontier AI Developmentcited by 10
34 more
  1. 2025The Singapore Consensus on Global AI Safety Research Prioritiescited by 9
  2. 2025The 2025 Foundation Model Transparency Indexcited by 9
  3. 2024Towards a Framework for Openness in Foundation Models: Proceedings from the Columbia Convening on Openness in Artificial Intelligencecited by 7
  4. 2025In-House Evaluation Is Not Enough: Towards Robust Third-Party Flaw Disclosure for General-Purpose AIcited by 7
  5. 2025The Accountability Paradox: How Platform API Restrictions Undermine AI Transparency Mandatescited by 7
  6. 2024Unveiling the Misuse Potential of Base Large Language Models via In-Context Learningcited by 5
  7. 2024To Err is AI : A Case Study Informing LLM Flaw Reporting Practicescited by 5
  8. 2025Real-World Gaps in AI Governance Researchcited by 5
  9. 2024Evaluating and Safeguarding the Adversarial Robustness of Retrieval-Based In-Context Learningcited by 4
  10. 2024Jailbreaking LLM-Controlled Robotscited by 4
  11. 2025Position: Ensuring mutual privacy is necessary for effective external evaluation of proprietary AI systemscited by 4
  12. 2025Who Evaluates AI's Social Impacts? Mapping Coverage and Gaps in First and Third Party Evaluationscited by 4
  13. 2026State-Dependent Safety Failures in Multi-Turn Language Model Interactioncited by 4
  14. 2024A Framework for Evaluating LLMs Under Task Indeterminacycited by 3
  15. 2025NeurIPS should lead scientific consensus on AI policycited by 3
  16. 2026"Unlimited Realm of Exploration and Experimentation": Methods and Motivations of AI-Generated Sexual Content Creatorscited by 3
  17. 2024Watermarking Large Language Models and the Generated Content: Opportunities and Challengescited by 2
  18. 2024Risk Sources and Risk Management Measures in Support of Standards for General-Purpose AI Systemscited by 2
  19. 2025The Pitfalls of "Security by Obscurity" And What They Mean for Transparent AIcited by 2
  20. 2025AI Agents Should be Regulated Based on the Extent of Their Autonomous Operationscited by 2
  21. 2025When Testing AI Tests Us: Safeguarding Mental Health on the Digital Frontlinescited by 2
  22. 2025GuardVal: Dynamic Large Language Model Jailbreak Evaluation for Comprehensive Safety Testingcited by 2
  23. 2025Audits Under Resource, Data, and Access Constraints: Scaling Laws For Less Discriminatory Alternativescited by 2
  24. 2025From Firewalls to Frontiers: AI Red-Teaming is a Domain-Specific Evolution of Cyber Red-Teamingcited by 2
  25. 2026Red-Teaming Agent Execution Contexts: Open-World Security Evaluation on OpenClawcited by 2
  26. 2024Bridging Today and the Future of Humanity: AI Safety in 2024 and Beyondcited by 1
  27. 2025Enabling External Scrutiny of AI Systems with Privacy-Enhancing Technologiescited by 1
  28. 2025Beyond Release: Access Considerations for Generative AI Systemscited by 1
  29. 2025When Search Goes Wrong: Red-Teaming Web-Augmented Large Language Modelscited by 1
  30. 2025Ask What Your Country Can Do For You: Towards a Public Red Teaming Modelcited by 1
  31. 2025I Stolenly Swear That I Am Up to (No) Good: Design and Evaluation of Model Stealing Attackscited by 0
  32. 2026Red Teaming LLMs as Socio-Technical Practice: From Exploration and Data Creation to Evaluationcited by 0
  33. 2026ARMOR 2025: A Military-Aligned Benchmark for Evaluating Large Language Model Safety Beyond Civilian Contextscited by 0
  34. 2026FLARE-AI: Flaw Reporting for AIcited by 0