mlsanityData to 14 Oct 2026

Evaluating the efficacy of LLM Safety Solutions : The Palit Benchmark Dataset

arXiv 2505.13028 ↗First posted 19 May 2025cs.CR cs.AI cs.CLcited by 2

On the map: LLM safety and security → Prompt injection and agent security

Abstract (arXiv)

Large Language Models (LLMs) are increasingly integrated into critical systems in industries like healthcare and finance. Users can often submit queries to LLM-enabled chatbots, some of which can enrich responses with information retrieved from internal databases storing sensitive data. This gives rise to a range of attacks in which a user submits a malicious query and the LLM-system outputs a response that creates harm to the owner, such as leaking internal data or creating legal liability by harming a third-party. While security tools are being developed to counter these threats, there is little formal evaluation of their effectiveness and usability. This study addresses this gap by conducting a thorough comparative analysis of LLM security tools. We identified 13 solutions (9 closed-source, 4 open-source), but only 7 were evaluated due to a lack of participation by proprietary model owners.To evaluate, we built a benchmark dataset of malicious prompts, and evaluate these tools performance against a baseline LLM model (ChatGPT-3.5-Turbo). Our results show that the baseline model has too many false positives to be used for this task. Lakera Guard and ProtectAI LLM Guard emerged as the best overall tools showcasing the tradeoff between usability and performance. The study concluded with recommendations for greater transparency among closed source providers, improved context-aware detections, enhanced open-source engagement, increased user awareness, and the adoption of more representative performance metrics.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 30 of its 70 references that are in the map. First posted on arXiv 19 May 2025.

Builds on

30 of 70 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024garak: A Framework for Security Probing Large Language Models16 Jun
  2. 2024InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents5 Mar
  3. 2024Watch Out for Your Agents! Investigating Backdoor Threats to LLM-Based Agents17 Feb
  4. 2023Summon a Demon and Bind it: A Grounded Theory of LLM Red Teaming10 Nov
  5. 2023Prompt Injection attack against LLM-integrated Applications8 Jun
  6. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection23 Feb

Other references · 24

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2022Training language models to follow instructions with human feedbackcited by 15,971
  5. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  6. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  7. 2023BloombergGPT: A Large Language Model for Financecited by 939
  8. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 832
  9. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 785
  10. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
  11. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 560
  12. 2018Poison Frogs! Targeted Clean-Label Poisoning Attacks on Neural Networkscited by 519
12 more
  1. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 489
  2. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 387
  3. 2020Weight Poisoning Attacks on Pre-trained Modelscited by 341
  4. 2023Large Language Model Alignment: A Surveycited by 243
  5. 2023Prompt as Triggers for Backdoor Attack: Examining the Vulnerability in Language Modelscited by 94
  6. 2023Conversational Health Agents: A Personalized LLM-Powered Agent Frameworkcited by 92
  7. 2022"Real Attackers Don't Compute Gradients": Bridging the Gap Between Adversarial ML Research and Practicecited by 56
  8. 2021How Should Pre-Trained Language Models Be Fine-Tuned Towards Adversarial Robustness?cited by 49
  9. 2023PoisonPrompt: Backdoor Attack on Prompt-based Large Language Modelscited by 42
  10. 2023A LLM Assisted Exploitation of AI-Guardiancited by 15
  11. 2023Watch Your Language: Investigating Content Moderation with Large Language Modelscited by 14
  12. 2023Punctuation Matters! Stealthy Backdoor Attack for Language Modelscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026You Told Me to Do It: Measuring Instructional Text-induced Private Data Leakage in LLM Agentscited by 3
  2. 2026Gate AI: LLM Security Benchmark Evaluation Methodology and Resultscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025Analysing Safety Risks in LLMs Fine-Tuned with Pseudo-Malicious Cyber Security Data
  2. 2025Safety and Security Analysis of Large Language Models: Benchmarking Risk Profile and Harm Potential
  3. 2025OffTopicEval: When Large Language Models Enter the Wrong Chat, Almost Always!
  4. 2024Fine-Tuning, Quantization, and LLMs: Navigating Unintended Outcomes
  5. 2025LLM-Safety Evaluations Lack Robustness
  6. 2025Data Security in Large Language Models: Risks, Defense, and Directions
  7. 2024Trust & Safety of LLMs and LLMs in Trust & Safety
  8. 2025aiXamine: Simplified LLM Safety and Security