mlsanityData to 14 Oct 2026

Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignment

arXiv 2308.09662 ↗First posted 18 Aug 2023cs.CLcited by 223

On the map: LLM safety and security → Prompt injection and agent security

Abstract (arXiv)

Larger language models (LLMs) have taken the world by storm with their massive multi-tasking capabilities simply by optimizing over a next-word prediction objective. With the emergence of their properties and encoded knowledge, the risk of LLMs producing harmful outputs increases, making them unfit for scalable deployment for the public. In this work, we propose a new safety evaluation benchmark RED-EVAL that carries out red-teaming. We show that even widely deployed models are susceptible to the Chain of Utterances-based (CoU) prompting, jailbreaking closed source LLM-based systems such as GPT-4 and ChatGPT to unethically respond to more than 65% and 73% of harmful queries. We also demonstrate the consistency of the RED-EVAL across 8 open-source LLMs in generating harmful responses in more than 86% of the red-teaming attempts. Next, we propose RED-INSTRUCT--An approach for the safety alignment of LLMs. It constitutes two phases: 1) HARMFULQA data collection: Leveraging CoU prompting, we collect a dataset that consists of 1.9K harmful questions covering a wide range of topics, 9.5K safe and 7.3K harmful conversations from ChatGPT; 2) SAFE-ALIGN: We demonstrate how the conversational dataset can be used for the safety alignment of LLMs by minimizing the negative log-likelihood over helpful responses and penalizing over harmful responses by gradient accent over sample loss. Our model STARLING, a fine-tuned Vicuna-7B, is observed to be more safely aligned when evaluated on RED-EVAL and HHH benchmarks while preserving the utility of the baseline models (TruthfulQA, MMLU, and BBH).

Key facts. Cited by 223 papers in the map (data through 14 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 23 of its 34 references that are in the map. First posted on arXiv 18 Aug 2023.

Builds on

23 of 34 references.

References · 23

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  6. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  7. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  8. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  9. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  10. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  11. 2020Aligning AI With Shared Human Valuescited by 723
  12. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 701
11 more
  1. 2018Gender Bias in Coreference Resolutioncited by 442
  2. 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervisioncited by 318
  3. 2022On Second Thought, Let's Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoningcited by 194
  4. 2021Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasetscited by 181
  5. 2023The Capacity for Moral Self-Correction in Large Language Modelscited by 173
  6. 2023Does Synthetic Data Generation of LLMs Help Clinical Text Mining?cited by 154
  7. 2022Legal Prompt Engineering for Multilingual Legal Judgement Predictioncited by 55
  8. 2023On the Educational Impact of ChatGPT: Is Artificial Intelligence Ready to Obtain a University Degree?cited by 48
  9. 2023Putting ChatGPT's Medical Advice to the (Turing) Testcited by 24
  10. 2023Beyond Classification: Financial Reasoning in State-of-the-Art Language Modelscited by 19
  11. 2023Towards Safer Generative Language Models: A Survey on Safety Risks, Evaluations, and Improvementscited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 20; 2024: 89; 2025: 72; 2026: 42.

223 papers.

Sort:

By area:

  1. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attackscited by 469
  2. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 461
  3. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Modelscited by 297
  4. 2024R-Judge: Benchmarking Safety Risk Awareness for LLM Agentscited by 288
  5. 2023Multilingual Jailbreak Challenges in Large Language Modelscited by 240
  6. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Surveycited by 230
  7. 2024SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Modelscited by 228
  8. 2023Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attackscited by 194
  9. 2024AEGIS: Online Adaptive AI Content Safety Moderation with Ensemble of LLM Expertscited by 136
  10. 2025Aegis2.0: A Diverse AI Safety Dataset and Risks Taxonomy for Alignment of LLM Guardrailscited by 132
  11. 2024Attacks, Defenses and Evaluations for LLM Conversation Safety: A Surveycited by 130
  12. 2025Large Language Models Often Know When They Are Being Evaluatedcited by 128
38 more
  1. 2024Language Models are Homer Simpson! Safety Re-Alignment of Fine-tuned Language Models through Task Arithmeticcited by 112
  2. 2024Weak-to-Strong Jailbreaking on Large Language Modelscited by 107
  3. 2024Aligner: Efficient Alignment by Learning to Correctcited by 98
  4. 2024Red-Teaming for Generative AI: Silver Bullet or Security Theater?cited by 83
  5. 2024FOFO: A Benchmark to Evaluate LLMs' Format-Following Capabilitycited by 80
  6. 2024SafetyPrompts: a Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safetycited by 77
  7. 2024JailbreakZoo: Survey, Landscapes, and Horizons in Jailbreaking Large Language and Vision-Language Modelscited by 70
  8. 2025STAR-1: Safer Alignment of Reasoning LLMs with 1K Datacited by 68
  9. 2023The Art of Defending: A Systematic Evaluation and Analysis of LLM Defense Strategies on Safety and Over-Defensivenesscited by 67
  10. 2024Fight Back Against Jailbreaking via Prompt Adversarial Tuningcited by 67
  11. 2024Uncovering Safety Risks of Large Language Models through Concept Activation Vectorcited by 67
  12. 2024T2VSafetyBench: Evaluating the Safety of Text-to-Video Generative Modelscited by 66
  13. 2024Red Teaming Visual Language Modelscited by 65
  14. 2024Navigating the Risks: A Survey of Security, Privacy, and Ethics Threats in LLM-Based Agentscited by 65
  15. 2024Aligning Large Language Models with Representation Editing: A Control Perspectivecited by 64
  16. 2024Introducing v0.5 of the AI Safety Benchmark from MLCommonscited by 62
  17. 2024SG-Bench: Evaluating LLM Safety Generalization Across Diverse Tasks and Prompt Typescited by 60
  18. 2024Multi-Turn Context Jailbreak Attack on Large Language Models From First Principlescited by 56
  19. 2024MultiTrust: A Comprehensive Benchmark Towards Trustworthy Multimodal Large Language Modelscited by 53
  20. 2023Privacy in Large Language Models: Attacks, Defenses and Future Directionscited by 51
  21. 2024Bag of Tricks: Benchmarking of Jailbreak Attacks on LLMscited by 51
  22. 2024Risks of AI Scientists: Prioritizing Safeguarding Over Autonomycited by 49
  23. 2024Beyond Chain-of-Thought: A Survey of Chain-of-X Paradigms for LLMscited by 48
  24. 2024ALI-Agent: Assessing LLMs' Alignment with Human Values via Agent-based Evaluationcited by 47
  25. 2024When LLM Meets DRL: Advancing Jailbreaking Efficiency via DRL-guided Searchcited by 47
  26. 2024JAILJUDGE: A Comprehensive Jailbreak Judge Benchmark with Multi-Agent Enhanced Explanation Evaluation Frameworkcited by 47
  27. 2024Self-Alignment of Large Language Models via Monopolylogue-based Social Scene Simulationcited by 46
  28. 2024Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)cited by 46
  29. 2024MedSafetyBench: Evaluating and Improving the Medical Safety of Large Language Modelscited by 44
  30. 2024Arondight: Red Teaming Large Vision Language Models with Auto-generated Multi-modal Jailbreak Promptscited by 42
  31. 2024A safety realignment framework via subspace-oriented model fusion for large language modelscited by 41
  32. 2023Stealthy and Persistent Unalignment on Large Language Models via Backdoor Injectionscited by 39
  33. 2024Securing Large Language Models: Threats, Vulnerabilities and Responsible Practicescited by 37
  34. 2024Leveraging the Context through Multi-Round Interactions for Jailbreaking Attackscited by 36
  35. 2024Assessment of Multimodal Large Language Models in Alignment with Human Valuescited by 36
  36. 2023Prompt Packer: Deceiving LLMs through Compositional Instruction with Hidden Attackscited by 35
  37. 2023Beyond Imitation: Leveraging Fine-grained Quality Signals for Alignmentcited by 35
  38. 2024Jailbreaking and Mitigation of Vulnerabilities in Large Language Modelscited by 34

50 most cited of 223 All 223 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026A Red Teaming Framework for Large Language Models: A Case Study on Faithfulness Evaluation
  2. 2024Code-Switching Red-Teaming: LLM Evaluation for Safety and Multilingual Understanding
  3. 2024ALERT: A Comprehensive Benchmark for Assessing Large Language Models' Safety through Red Teaming
  4. 2025TRIDENT: Enhancing Large Language Model Safety with Tri-Dimensional Diversified Red-Teaming Data Synthesis
  5. 2025Multi-lingual Multi-turn Automated Red Teaming for LLMs
  6. 2023Explore, Establish, Exploit: Red Teaming Language Models from Scratch
  7. 2023MART: Improving LLM Safety with Multi-round Automatic Red-Teaming
  8. 2023All Languages Matter: On the Multilingual Safety of Large Language Models