# Current state of LLM Risks and AI Guardrails

arXiv: https://arxiv.org/abs/2406.12934 · HTML: https://mlsanity.com/paper/2406.12934/

## Key facts

- Cited by 46 papers in the map (data through 14 Oct 2026).
- Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security.
- Builds on 24 of its 44 references that are in the map.
- First posted on arXiv 16 Jun 2024.

## Placed in

- LLM safety and security / [Safety alignment, refusal and guardrails](https://mlsanity.com/safety/safety-alignment-refusal-and-guardrails/)

## Builds on (most cited first, 24 of 24)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Membership Inference Attacks against Machine Learning Models](https://mlsanity.com/paper/1610.05820/) (2016)
- [Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](https://mlsanity.com/paper/2202.12837/) (2022)
- [Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations](https://mlsanity.com/paper/2312.06674/) (2023)
- [StereoSet: Measuring stereotypical bias in pretrained language models](https://mlsanity.com/paper/2004.09456/) (2020)
- [Gender Bias in Coreference Resolution](https://mlsanity.com/paper/1804.09301/) (2018)
- [NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Rails](https://mlsanity.com/paper/2310.10501/) (2023)
- [ToolQA: A Dataset for LLM Question Answering with External Tools](https://mlsanity.com/paper/2306.13304/) (2023)
- [Universal Self-Consistency for Large Language Model Generation](https://mlsanity.com/paper/2311.17311/) (2023)
- [Adversarial Robustness: From Self-Supervised Pre-Training to Fine-Tuning](https://mlsanity.com/paper/2003.12862/) (2020)
- [Harms from Increasingly Agentic Algorithmic Systems](https://mlsanity.com/paper/2302.10329/) (2023)
- [DECAF: Generating Fair Synthetic Data Using Causally-Aware Generative Networks](https://mlsanity.com/paper/2110.12884/) (2021)
- [MatPlotAgent: Method and Evaluation for LLM-Based Agentic Scientific Data Visualization](https://mlsanity.com/paper/2402.11453/) (2024)
- [Systematic Biases in LLM Simulations of Debates](https://mlsanity.com/paper/2402.04049/) (2024)
- [Unfamiliar Finetuning Examples Control How Language Models Hallucinate](https://mlsanity.com/paper/2403.05612/) (2024)
- [Reliable, Adaptable, and Attributable Language Models with Retrieval](https://mlsanity.com/paper/2403.03187/) (2024)
- [When "A Helpful Assistant" Is Not Really Helpful: Personas in System Prompts Do Not Improve Performances of Large Language Models](https://mlsanity.com/paper/2311.10054/) (2023)
- [Prompt Injection: Parameterization of Fixed Inputs](https://mlsanity.com/paper/2206.11349/) (2022)
- [FairRAG: Fair Human Generation via Fair Retrieval Augmentation](https://mlsanity.com/paper/2403.19964/) (2024)
- [Domain Adaptive Transfer Learning with Specialist Models](https://mlsanity.com/paper/1811.07056/) (2018)
- [Knowledge Graphs as Context Sources for LLM-Based Explanations of Learning Recommendations](https://mlsanity.com/paper/2403.03008/) (2024)
- [A Simple Fine-tuning Is All You Need: Towards Robust Deep Learning Via Adversarial Fine-tuning](https://mlsanity.com/paper/2012.13628/) (2020)
- [In-context learning agents are asymmetric belief updaters](https://mlsanity.com/paper/2402.03969/) (2024)
- [StablePT: Towards Stable Prompting for Few-shot Learning via Input Separation](https://mlsanity.com/paper/2404.19335/) (2024)

## Built on by (most cited first, 30 of 46)

- [Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Survey](https://mlsanity.com/paper/2409.18169/) (2024)
- [Large Language Model Safety: A Holistic Survey](https://mlsanity.com/paper/2412.17686/) (2024)
- [LLMs are Also Effective Embedding Models: An In-depth Overview](https://mlsanity.com/paper/2412.12591/) (2024)
- [Agentic AI Security: Threats, Defenses, Evaluation, and Open Challenges](https://mlsanity.com/paper/2510.23883/) (2025)
- [Octopus: Alleviating Hallucination via Dynamic Contrastive Decoding](https://mlsanity.com/paper/2503.00361/) (2025)
- [Attack Atlas: A Practitioner's Perspective on Challenges and Pitfalls in Red Teaming GenAI](https://mlsanity.com/paper/2409.15398/) (2024)
- [A Framework for Formalizing LLM Agent Security](https://mlsanity.com/paper/2603.19469/) (2026)
- [Swiss Cheese Model for AI Safety: A Taxonomy and Reference Architecture for Multi-Layered Guardrails of Foundation Model Based Agents](https://mlsanity.com/paper/2408.02205/) (2024)
- [Agentic AI Systems Applied to tasks in Financial Services: Modeling and model risk management crews](https://mlsanity.com/paper/2502.05439/) (2025)
- [DualBreach: Efficient Dual-Jailbreaking via Target-Driven Initialization and Multi-Target Optimization](https://mlsanity.com/paper/2504.18564/) (2025)
- [No Free Lunch with Guardrails](https://mlsanity.com/paper/2504.00441/) (2025)
- [From Vulnerabilities to Remediation: A Systematic Literature Review of LLMs in Code Security](https://mlsanity.com/paper/2412.15004/) (2024)
- [Interpretation Meets Safety: A Survey on Interpretation Methods and Tools for Improving LLM Safety](https://mlsanity.com/paper/2506.05451/) (2025)
- [Language Models Can Predict Their Own Behavior](https://mlsanity.com/paper/2502.13329/) (2025)
- [MOCHA: Are Code Language Models Robust Against Multi-Turn Malicious Coding Prompts?](https://mlsanity.com/paper/2507.19598/) (2025)
- [Anyone Can Jailbreak: Prompt-Based Attacks on LLMs and T2Is](https://mlsanity.com/paper/2507.21820/) (2025)
- [Building a Domain-specific Guardrail Model in Production](https://mlsanity.com/paper/2408.01452/) (2024)
- [Accidental Vulnerability: Factors in Fine-Tuning that Shift Model Safeguards](https://mlsanity.com/paper/2505.16789/) (2025)
- [OpenRAG: Optimizing RAG End-to-End via In-Context Retrieval Learning](https://mlsanity.com/paper/2503.08398/) (2025)
- [Bridging Industrial Expertise and XR with LLM-Powered Conversational Agents](https://mlsanity.com/paper/2504.05527/) (2025)
- [Uncovering the Fragility of Trustworthy LLMs through Chinese Textual Ambiguity](https://mlsanity.com/paper/2507.23121/) (2025)
- [Pet-Bench: Benchmarking the Abilities of Large Language Models as E-Pets in Social Network Services](https://mlsanity.com/paper/2506.03761/) (2025)
- [H2HTalk: Evaluating Large Language Models as Emotional Companion](https://mlsanity.com/paper/2507.03543/) (2025)
- [TopoPilot: Reliable Conversational Workflow Automation for Topological Data Analysis and Visualization](https://mlsanity.com/paper/2603.25063/) (2026)
- [From Cool Demos to Production-Ready FMware: Core Challenges and a Technology Roadmap](https://mlsanity.com/paper/2410.20791/) (2024)
- [Libra-Leaderboard: Towards Responsible AI through a Balanced Leaderboard of Safety and Capability](https://mlsanity.com/paper/2412.18551/) (2024)
- [Scam2Prompt: A Scalable Framework for Auditing Malicious Scam Endpoints in Production LLMs](https://mlsanity.com/paper/2509.02372/) (2025)
- [Proactive Hardening of LLM Defenses with HASTE](https://mlsanity.com/paper/2601.19051/) (2026)
- [Bielik Guard: Efficient Polish Language Safety Classifiers for LLM Content Moderation](https://mlsanity.com/paper/2602.07954/) (2026)
- [Bridging Today and the Future of Humanity: AI Safety in 2024 and Beyond](https://mlsanity.com/paper/2410.18114/) (2024)

## Abstract (arXiv)

Large language models (LLMs) have become increasingly sophisticated, leading to widespread deployment in sensitive applications where safety and reliability are paramount. However, LLMs have inherent risks accompanying them, including bias, potential for unsafe actions, dataset poisoning, lack of explainability, hallucinations, and non-reproducibility. These risks necessitate the development of "guardrails" to align LLMs with desired behaviors and mitigate potential harm. This work explores the risks associated with deploying LLMs and evaluates current approaches to implementing guardrails and model alignment techniques. We examine intrinsic and extrinsic bias evaluation methods and discuss the importance of fairness metrics for responsible AI development. The safety and reliability of agentic LLMs (those capable of real-world actions) are explored, emphasizing the need for testability, fail-safes, and situational awareness. Technical strategies for securing LLMs are presented, including a layered protection model operating at external, secondary, and internal levels. System prompts, Retrieval-Augmented Generation (RAG) architectures, and techniques to minimize bias and protect privacy are highlighted. Effective guardrail design requires a deep understanding of the LLM's intended use case, relevant regulations, and ethical considerations. Striking a balance between competing requirements, such as accuracy and privacy, remains an ongoing challenge. This work underscores the importance of continuous research and development to ensure the safe and responsible use of LLMs in real-world applications.