mlsanityData to 7 Oct 2026

Embedding-based classifiers can detect prompt injection attacks

arXiv 2410.22284 ↗First posted 29 Oct 2024cs.CR cs.LGcited by 24

On the map: LLM safety and security → Prompt injection and agent security

Abstract (arXiv)

Large Language Models (LLMs) are seeing significant adoption in every type of organization due to their exceptional generative capabilities. However, LLMs are found to be vulnerable to various adversarial attacks, particularly prompt injection attacks, which trick them into producing harmful or inappropriate content. Adversaries execute such attacks by crafting malicious prompts to deceive the LLMs. In this paper, we propose a novel approach based on embedding-based Machine Learning (ML) classifiers to protect LLM-based applications against this severe threat. We leverage three commonly used embedding models to generate embeddings of malicious and benign prompts and utilize ML classifiers to predict whether an input prompt is malicious. Out of several traditional ML methods, we achieve the best performance with classifiers built using Random Forest and XGBoost. Our classifiers outperform state-of-the-art prompt injection classifiers available in open-source implementations, which use encoder-only neural networks.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 29 of its 45 references that are in the map. First posted on arXiv 29 Oct 2024.

Builds on

29 of 45 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Prompts26 Feb
  2. 2024StruQ: Defending Against Prompt Injection with Structured Queries9 Feb
  3. 2023Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Models21 Dec
  4. 2023Assessing Prompt Injection Risks in 200+ Custom GPTs20 Nov
  5. 2023Prompt Injection attack against LLM-integrated Applications8 Jun
  6. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection23 Feb
  7. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned23 Aug

Other references · 22

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,120
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  4. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,841
  5. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  6. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  7. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  8. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 760
  9. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
  10. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 579
  11. 2023Detecting Language Model Attacks with Perplexitycited by 411
  12. 2019Linguistic Knowledge and Transferability of Contextual Representationscited by 382
10 more
  1. 2022Text and Code Embeddings by Contrastive Pre-Trainingcited by 358
  2. 2023Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attackscited by 268
  3. 2013Minimizing Finite Sums with the Stochastic Average Gradientcited by 164
  4. 2023BooookScore: A systematic exploration of book-length summarization in the era of LLMscited by 137
  5. 2022Efficient Few-Shot Learning Without Promptscited by 135
  6. 2023The Science of Detecting LLM-Generated Textscited by 126
  7. 2023Enhancing Financial Sentiment Analysis via Retrieval Augmented Large Language Modelscited by 67
  8. 2022"Real Attackers Don't Compute Gradients": Bridging the Gap Between Adversarial ML Research and Practicecited by 55
  9. 2023Exploring Human-Like Translation Strategy with Large Language Modelscited by 44
  10. 2024Special Characters Attack: Toward Scalable Training Data Extraction From Large Language Modelscited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 14; 2026: 10.

24 papers.

Sort:

By area:

  1. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  2. 2025Securing AI Agents with Information-Flow Controlcited by 108
  3. 2025A Survey on Trustworthy LLM Agents: Threats and Countermeasurescited by 93
  4. 2025A Critical Evaluation of Defenses against Prompt Injection Attackscited by 31
  5. 2025Security Concerns for Large Language Models: A Surveycited by 26
  6. 2025WAInjectBench: Benchmarking Prompt Injection Detections for Web Agentscited by 21
  7. 2025CASE-Bench: Context-Aware SafEty Benchmark for Large Language Modelscited by 17
  8. 2025SoK: Trust-Authorization Mismatch in LLM Agent Interactionscited by 11
  9. 2026WebSentinel: Detecting and Localizing Prompt Injection Attacks for Web Agentscited by 10
  10. 2026Optimizing Agent Planning for Security and Autonomycited by 10
  11. 2025Gandalf the Red: Adaptive Security for LLMscited by 7
  12. 2025Evaluating the Robustness of Retrieval-Augmented Generation to Adversarial Evidence in the Health Domaincited by 5
12 more
  1. 2025Efficient LLM Moderation with Multi-Layer Latent Prototypescited by 4
  2. 2026SnapGuard: Lightweight Prompt Injection Detection for Screenshot-Based Web Agentscited by 3
  3. 2025CachePrune: Teaching LLMs What Not to Follow via KV-Cache Editingcited by 2
  4. 2025Helping Large Language Models Protect Themselves: An Enhanced Filtering and Summarization Systemcited by 2
  5. 2026Zero-Shot Embedding Drift Detection: A Lightweight Defense Against Prompt Injections in LLMscited by 2
  6. 2026Enhancing Safety of Large Language Models via Embedding Space Separationcited by 2
  7. 2025AegisMCP: Online Graph Intrusion Detection for Tool-Augmented LLMs on Edge Devicescited by 1
  8. 2026Prompt Attack Detection with LLM-as-a-Judge and Mixture-of-Modelscited by 1
  9. 2026Informationally Compressive Anonymization: Non-Degrading Sensitive Input Protection for Privacy-Preserving Supervised Machine Learningcited by 0
  10. 2026PROMPT2BOX:Improving LLM Weakness Discovery and Specificity Estimation by Uncovering Entailment Structure among Promptscited by 0
  11. 2026Cross-Lingual Jailbreak Detection via Semantic Codebookscited by 0
  12. 2026PsychoPass: Geometric Profiling of Multi-Turn Adversarial LLM Conversationscited by 0