mlsanityData to 7 Oct 2026

RigorLLM: Resilient Guardrails for Large Language Models against Undesired Content

arXiv 2403.13031 ↗First posted 19 Mar 2024cs.CR cs.AI cs.CLcited by 65

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

Recent advancements in Large Language Models (LLMs) have showcased remarkable capabilities across various tasks in different domains. However, the emergence of biases and the potential for generating harmful content in LLMs, particularly under malicious inputs, pose significant challenges. Current mitigation strategies, while effective, are not resilient under adversarial attacks. This paper introduces Resilient Guardrails for Large Language Models (RigorLLM), a novel framework designed to efficiently and effectively moderate harmful and unsafe inputs and outputs for LLMs. By employing a multi-faceted approach that includes energy-based training data augmentation through Langevin dynamics, optimizing a safe suffix for inputs via minimax optimization, and integrating a fusion-based model combining robust KNN with LLMs based on our data augmentation, RigorLLM offers a robust solution to harmful content moderation. Our experimental evaluations demonstrate that RigorLLM not only outperforms existing baselines like OpenAI API and Perspective API in detecting harmful content but also exhibits unparalleled resilience to jailbreaking attacks. The innovative use of constrained optimization and a fusion-based guardrail approach represents a significant step forward in developing more secure and reliable LLMs, setting a new standard for content moderation frameworks in the face of evolving digital threats.

Key facts. Cited by 65 papers in the map (data through 7 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 34 of its 37 references that are in the map. First posted on arXiv 19 Mar 2024.

Builds on

34 of 37 references.

Closest ancestors · 11

Same direction, newest first

  1. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs12 Jan
  2. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automatically4 Dec
  3. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation10 Oct
  4. 2023Multilingual Jailbreak Challenges in Large Language Models10 Oct
  5. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations10 Oct
  6. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks5 Oct
  7. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models3 Oct
  8. 2023Low-Resource Languages Jailbreak GPT-43 Oct
  9. 2023Defending Against Alignment-Breaking Attacks via Robustly Aligned LLM18 Sep
  10. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher12 Aug
  11. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul

Other references · 23

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  7. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  8. 2023Mistral 7Bcited by 3,065
  9. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  10. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  11. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  12. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
11 more
  1. 2023DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Modelscited by 505
  2. 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Trainingcited by 478
  3. 2023NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Railscited by 347
  4. 2022A Holistic Approach to Undesired Content Detection in the Real Worldcited by 337
  5. 2023ToxicChat: Unveiling Hidden Challenges of Toxicity Detection in Real-World User-AI Conversationcited by 224
  6. 2023Cross-Entropy Loss Functions: Theoretical Analysis and Applicationscited by 206
  7. 2022A New Generation of Perspective API: Efficient Multilingual Character-level Transformerscited by 181
  8. 2022COLD Decoding: Energy-based Constrained Text Generation with Langevin Dynamicscited by 159
  9. 2023Adversarial Demonstration Attacks on Large Language Modelscited by 58
  10. 2017Analyzing the Robustness of Nearest Neighbors to Adversarial Examplescited by 36
  11. 2023Make Them Spill the Beans! Coercive Knowledge Extraction from (Production) LLMscited by 26

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 27; 2025: 26; 2026: 12.

65 papers.

Sort:

By area:

  1. 2024AmpleGCG: Learning a Universal and Transferable Generative Model of Adversarial Suffixes for Jailbreaking Both Open and Closed LLMscited by 160
  2. 2024Tamper-Resistant Safeguards for Open-Weight LLMscited by 132
  3. 2024AEGIS: Online Adaptive AI Content Safety Moderation with Ensemble of LLM Expertscited by 126
  4. 2024Llama Guard 3 Vision: Safeguarding Human-AI Image Understanding Conversationscited by 118
  5. 2024Know Your Limits: A Survey of Abstention in Large Language Modelscited by 116
  6. 2024GuardAgent: Safeguard LLM Agents by a Guard Agent via Knowledge-Enabled Reasoningcited by 113
  7. 2024Large Language Model Unlearning via Embedding-Corrupted Promptscited by 112
  8. 2025GuardReasoner: Towards Reasoning-based LLM Safeguardscited by 89
  9. 2025AGrail: A Lifelong Agent Guardrail with Effective and Adaptive Safety Detectioncited by 82
  10. 2024LLMs know their vulnerabilities: Uncover Safety Gaps through Natural Distribution Shiftscited by 80
  11. 2024Improved Few-Shot Jailbreaking Can Circumvent Aligned Language Models and Their Defensescited by 68
  12. 2024Refuse Whenever You Feel Unsafe: Improving Safety in LLMs via Decoupled Refusal Trainingcited by 67
38 more
  1. 2024Bag of Tricks: Benchmarking of Jailbreak Attacks on LLMscited by 47
  2. 2024Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)cited by 44
  3. 2024JAILJUDGE: A Comprehensive Jailbreak Judge Benchmark with Multi-Agent Enhanced Explanation Evaluation Frameworkcited by 44
  4. 2024Model Merging and Safety Alignment: One Bad Model Spoils the Bunchcited by 41
  5. 2024R2-Guard: Robust Reasoning Enabled LLM Guardrail via Knowledge-Enhanced Logical Reasoningcited by 41
  6. 2024A Survey of Useful LLM Evaluationcited by 28
  7. 2024JailbreakEval: An Integrated Toolkit for Evaluating Jailbreak Attempts Against Large Language Modelscited by 28
  8. 2024On Calibration of LLM-based Guard Models for Reliable Content Moderationcited by 21
  9. 2025Frontier AI's Impact on the Cybersecurity Landscapecited by 21
  10. 2025SoK: Evaluating Jailbreak Guardrails for Large Language Modelscited by 17
  11. 2025DynaGuard: A Dynamic Guardian Model With User-Defined Policiescited by 16
  12. 2024Mitigating Backdoor Threats to Large Language Models: Advancement and Challengescited by 14
  13. 2024Swiss Cheese Model for AI Safety: A Taxonomy and Reference Architecture for Multi-Layered Guardrails of Foundation Model Based Agentscited by 13
  14. 2024Benchmarking LLM Guardrails in Handling Multilingual Toxicitycited by 13
  15. 2025X-Guard: Multilingual Guard Agent for Content Moderationcited by 13
  16. 2025LLM Jailbreak Detection for (Almost) Free!cited by 13
  17. 2025SecurityLingua: Efficient Defense of LLM Jailbreak Attacks via Security-Aware Prompt Compressioncited by 12
  18. 2024STAND-Guard: A Small Task-Adaptive Content Moderation Modelcited by 11
  19. 2025ControlNET: A Firewall for RAG-based LLM Systemcited by 11
  20. 2025No Free Lunch with Guardrailscited by 10
  21. 2025Orchestrating Agents and Data for Enterprise: A Blueprint Architecture for Compound AIcited by 9
  22. 2025MrGuard: A Multilingual Reasoning Guardrail for Universal LLM Safetycited by 9
  23. 2024A Survey on Responsible Generative AI: What to Generate and What Notcited by 8
  24. 2024Survival of the Safest: Towards Secure Prompt Optimization through Interleaved Multi-Objective Evolutioncited by 8
  25. 2025Smoothed Embeddings for Robust Language Modelscited by 8
  26. 2025Improving Large Language Model Safety with Contrastive Representation Learningcited by 7
  27. 2025Jailbreak-Tuning: Models Efficiently Learn Jailbreak Susceptibilitycited by 6
  28. 2025Who's Asking? Investigating Bias Through the Lens of Disability Framed Queries in LLMscited by 6
  29. 2025Training-Free Policy Violation Detection via Activation-Space Whitening in LLMscited by 6
  30. 2024LLM Safeguard is a Double-Edged Sword: Exploiting False Positives for Denial-of-Service Attackscited by 5
  31. 2025Evaluating the Robustness of Retrieval-Augmented Generation to Adversarial Evidence in the Health Domaincited by 5
  32. 2025Understanding and Mitigating Risks of Generative AI in Financial Servicescited by 4
  33. 2025Co-Investigator AI: The Rise of Agentic AI for Smarter, Trustworthy AML Compliance Narrativescited by 4
  34. 2025SoK: Systematizing LLM Prompt Security: Taxonomies, Datasets, and Unified Evaluation of Attacks and Defensescited by 4
  35. 2025QuadSentinel: Sequent Safety for Machine-Checkable Control in Multi-agent Systemscited by 4
  36. 2024MoJE: Mixture of Jailbreak Experts, Naive Tabular Classifiers as Guard for Prompt Attackscited by 3
  37. 2024Buster: Implanting Semantic Backdoor into Text Encoder to Mitigate NSFW Content Generationcited by 3
  38. 2025Scaling Reinforcement Learning for Content Moderation with Large Language Modelscited by 3

50 most cited of 65 All 65 (JSON)