mlsanityData to 7 Oct 2026

Tiny Refinements Elicit Resilience: Toward Efficient Prefix-Model Against LLM Red-Teaming

arXiv 2405.12604 ↗First posted 21 May 2024cs.CL cs.AIcited by 6

On the map: LLM safety and security → Prompt injection and agent security

Abstract (arXiv)

With the proliferation of red-teaming strategies for Large Language Models (LLMs), the deficiency in the literature about improving the safety and robustness of LLM defense strategies is becoming increasingly pronounced. This paper introduces the LLM-based sentinel model as a plug-and-play prefix module designed to reconstruct the input prompt with just a few (<30) additional tokens, effectively reducing toxicity in responses from target LLMs. The sentinel model naturally overcomes the textitparameter inefficiency and textitlimited model accessibility for fine-tuning large target models. We employ an interleaved training regimen using Proximal Policy Optimization (PPO) to optimize both red team and sentinel models dynamically, incorporating a value head-sharing mechanism inspired by the multi-agent centralized critic to manage the complex interplay between agents. Our extensive experiments across text-to-text and text-to-image demonstrate the effectiveness of our approach in mitigating toxic outputs, even when dealing with larger models like textttLlama-2, textttGPT-3.5 and textttStable-Diffusion, highlighting the potential of our framework in enhancing safety and robustness in various applications.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 44 of its 54 references that are in the map. First posted on arXiv 21 May 2024.

Builds on

44 of 54 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Curiosity-driven Red-teaming for Large Language Models29 Feb
  2. 2023Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignment18 Aug
  3. 2023Explore, Establish, Exploit: Red Teaming Language Models from Scratch15 Jun
  4. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection23 Feb
  5. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned23 Aug
  6. 2022Red Teaming Language Models with Language Models7 Feb

Other references · 38

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2017Deep reinforcement learning from human preferencescited by 3,846
  7. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  8. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  9. 2020Learning to summarize from human feedbackcited by 2,562
  10. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  11. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
  12. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
26 more
  1. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  2. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 847
  3. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 795
  4. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 760
  5. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 733
  6. 2022Teaching Models to Express Their Uncertainty in Wordscited by 731
  7. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 695
  8. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
  9. 2020KILT: a Benchmark for Knowledge Intensive Language Taskscited by 571
  10. 2022ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detectioncited by 419
  11. 2023A Survey of Hallucination in Large Foundation Modelscited by 365
  12. 2023HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Modelscited by 329
  13. 2022Teaching language models to support answers with verified quotescited by 290
  14. 2023Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attackscited by 268
  15. 2022Red-Teaming the Stable Diffusion Safety Filtercited by 257
  16. 2020Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detectioncited by 217
  17. 2021Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasetscited by 179
  18. 2023ProPILE: Probing Privacy Leakage in Large Language Modelscited by 127
  19. 2023Black-Box Prompt Optimization: Aligning Large Language Models without Model Trainingcited by 122
  20. 2023FLIRT: Feedback Loop In-context Red Teamingcited by 81
  21. 2024Building Guardrails for Large Language Modelscited by 61
  22. 2023Learning to Generate Better Than Your LLMcited by 49
  23. 2021Mitigating harm in language models with conditional-likelihood filtrationcited by 38
  24. 2022Leashing the Inner Demons: Self-Detoxification for Language Modelscited by 19
  25. 2022You Are What You Write: Preserving Privacy in the Era of Large Language Modelscited by 11
  26. 2023Counterfactuals of Counterfactuals: a back-translation-inspired approach to analyse counterfactual editorscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 0; 2026: 2.

6 papers.

  1. 2024Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)cited by 44
  2. 2024BlueSuffix: Reinforced Blue Teaming for Vision-Language Models Against Jailbreak Attackscited by 28
  3. 2024Recent advancements in LLM Red-Teaming: Techniques, Defenses, and Ethical Considerationscited by 8
  4. 2026ProGRank: Probe-Gradient Reranking to Defend Dense-Retriever RAG from Corpus Poisoningcited by 2
  5. 2024Data Augmentation for Continual RL via Adversarial Gradient Episodic Memorycited by 0
  6. 2026A Cross-Architecture Audit of Direction-Based Inference-Time Defences in Vision-Language Modelscited by 0