RedBench: A Universal Dataset for Comprehensive Red Teaming of Large Language Models
On the map: LLM safety and security → Prompt injection and agent security
Abstract (arXiv)
As large language models (LLMs) become integral to safety-critical applications, ensuring their robustness against adversarial prompts is paramount. However, existing red teaming datasets suffer from inconsistent risk categorizations, limited domain coverage, and outdated evaluations, hindering systematic vulnerability assessments. To address these challenges, we introduce RedBench, a universal dataset aggregating 37 benchmark datasets from leading conferences and repositories, comprising 29,362 samples across attack and refusal prompts. RedBench employs a standardized taxonomy with 22 risk categories and 19 domains, enabling consistent and comprehensive evaluations of LLM vulnerabilities. We provide a detailed analysis of existing datasets, establish baselines for modern LLMs, and open-source the dataset and evaluation code. Our contributions facilitate robust comparisons, foster future research, and promote the development of secure and reliable LLMs for real-world deployment. Code: https://github.com/knoveleng/redeval
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 44 of its 65 references that are in the map. First posted on arXiv 7 Jan 2026.
Builds on
44 of 65 references.
Closest ancestors · 6
Same direction, newest first
- 2025RainbowPlus: Enhancing Adversarial Prompt Generation via Evolutionary Quality-Diversity Search21 Apr
- 2024Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Prompts26 Feb
- 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal6 Feb
- 2023Attack Prompt Generation for Red Teaming and Defending Large Language Models19 Oct
- 2023Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignment18 Aug
- 2022Red Teaming Language Models with Language Models7 Feb
Other references · 38
- 2024The Llama 3 Herd of Modelscited by 14,192
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
- 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
- 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
- 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
- 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
- 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
- 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 738
- 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 619
- 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 520
- 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 494
26 more
- 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 488
- 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 477
- 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 457
- 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 439
- 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 377
- 2024A StrongREJECT for Empty Jailbreakscited by 352
- 2023ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledgecited by 324
- 2024SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusalcited by 224
- 2024OR-Bench: An Over-Refusal Benchmark for Large Language Modelscited by 222
- 2022On Second Thought, Let's Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoningcited by 191
- 2024AutoDAN-Turbo: A Lifelong Agent for Strategy Self-Exploration to Jailbreak LLMscited by 185
- 2023Purple Llama CyberSecEval: A Secure Coding Benchmark for Language Modelscited by 130
- 2024Language Models are Homer Simpson! Safety Re-Alignment of Fine-tuned Language Models through Task Arithmeticcited by 105
- 2024The Art of Saying No: Contextual Noncompliance in Language Modelscited by 100
- 2022When to Make Exceptions: Exploring Language Models as Accounts of Human Moral Judgmentcited by 93
- 2021On the Safety of Conversational Models: Taxonomy, Dataset, and Benchmarkcited by 73
- 2024SG-Bench: Evaluating LLM Safety Generalization Across Diverse Tasks and Prompt Typescited by 58
- 2023Latent Jailbreak: A Benchmark for Evaluating Text Safety and Output Robustness of Large Language Modelscited by 56
- 2024CoSafe: Evaluating Large Language Model Safety in Multi-Turn Dialogue Coreferencecited by 51
- 2024MedSafetyBench: Evaluating and Improving the Medical Safety of Large Language Modelscited by 43
- 2022SafeText: A Benchmark for Exploring Physical Safety in Language Modelscited by 41
- 2023Training Models to Generate, Recognize, and Reframe Unhelpful Thoughtscited by 31
- 2024WalledEval: A Comprehensive Safety Evaluation Toolkit for Large Language Modelscited by 17
- 2024Gender Bias in Decision-Making with Large Language Models: A Study of Relationship Conflictscited by 15
- 2023Women Are Beautiful, Men Are Leaders: Gender Stereotypes in Machine Translation and Language Modelingcited by 12
- 2025Gandalf the Red: Adaptive Security for LLMscited by 7
Built on by
Citing papers per year: 2026: 3.
3 papers.