mlsanityData to 7 Oct 2026

Can LLMs be Fooled? Investigating Vulnerabilities in LLMs

arXiv 2407.20529 ↗First posted 30 Jul 2024cs.LG cs.CRcited by 5

Not on the map. Why?

Abstract (arXiv)

The advent of Large Language Models (LLMs) has garnered significant popularity and wielded immense power across various domains within Natural Language Processing (NLP). While their capabilities are undeniably impressive, it is crucial to identify and scrutinize their vulnerabilities especially when those vulnerabilities can have costly consequences. One such LLM, trained to provide a concise summarization from medical documents could unequivocally leak personal patient data when prompted surreptitiously. This is just one of many unfortunate examples that have been unveiled and further research is necessary to comprehend the underlying reasons behind such vulnerabilities. In this study, we delve into multiple sections of vulnerabilities which are model-based, training-time, inference-time vulnerabilities, and discuss mitigation strategies including "Model Editing" which aims at modifying LLMs behavior, and "Chroma Teaming" which incorporates synergy of multiple teaming strategies to enhance LLMs' resilience. This paper will synthesize the findings from each vulnerability section and propose new directions of research and development. By understanding the focal points of current vulnerabilities, we can better anticipate and mitigate future risks, paving the road for more robust and secure LLMs.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 59 of its 73 references that are in the map. First posted on arXiv 30 Jul 2024.

Builds on

59 of 73 references.

References · 59

  1. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  2. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  3. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
  4. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,339
  5. 2022Editing Models with Task Arithmeticcited by 1,076
  6. 2017Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoningcited by 1,012
  7. 2022Red Teaming Language Models with Language Modelscited by 983
  8. 2022Mass-Editing Memory in a Transformercited by 976
  9. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 760
  10. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  11. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
  12. 2021Fast Model Editing at Scalecited by 491
38 more
  1. 2022Memory-Based Model Editing at Scalecited by 443
  2. 2023Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defensecited by 401
  3. 2023Can AI-Generated Text be Reliably Detected?cited by 372
  4. 2023Editing Large Language Models: Problems, Methods, and Opportunitiescited by 367
  5. 2023Multi-step Jailbreaking Privacy Attacks on ChatGPTcited by 320
  6. 2023Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attackscited by 268
  7. 2023Poisoning Language Models During Instruction Tuningcited by 254
  8. 2023The False Promise of Imitating Proprietary LLMscited by 254
  9. 2020ONION: A Simple and Effective Defense Against Textual Backdoor Attackscited by 231
  10. 2023Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignmentcited by 213
  11. 2021Hidden Killer: Invisible Textual Backdoor Attacks with Syntactic Triggercited by 203
  12. 2020BadNL: Backdoor Attacks against NLP Models with Semantic-preserving Improvementscited by 191
  13. 2023Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attackscited by 189
  14. 2024Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Promptscited by 189
  15. 2023MART: Improving LLM Safety with Multi-round Automatic Red-Teamingcited by 161
  16. 2023Auditing large language models: a three-layered approachcited by 135
  17. 2023Purple Llama CyberSecEval: A Secure Coding Benchmark for Language Modelscited by 130
  18. 2023On the Risk of Misinformation Pollution with Large Language Modelscited by 124
  19. 2023Explore, Establish, Exploit: Red Teaming Language Models from Scratchcited by 111
  20. 2020Just How Toxic is Data Poisoning? A Unified Benchmark for Backdoor and Data Poisoning Attackscited by 109
  21. 2023EasyEdit: An Easy-to-use Knowledge Editing Framework for Large Language Modelscited by 97
  22. 2021Be Careful about Poisoned Word Embeddings: Exploring the Vulnerability of the Embedding Layers in NLP Modelscited by 96
  23. 2019DAWN: Dynamic Adversarial Watermarking of Neural Networkscited by 94
  24. 2023Red teaming ChatGPT via Jailbreaking: Bias, Robustness, Reliability and Toxicitycited by 86
  25. 2024Detoxifying Large Language Models via Knowledge Editingcited by 86
  26. 2023Use of LLMs for Illicit Purposes: Threats, Prevention Measures, and Vulnerabilitiescited by 76
  27. 2024Defending LLMs against Jailbreaking Attacks via Backtranslationcited by 65
  28. 2022BadPrompt: Backdoor Attacks on Continuous Promptscited by 59
  29. 2023Adversarial Demonstration Attacks on Large Language Modelscited by 58
  30. 2022Exploring the Universal Vulnerability of Prompt-based Learning Paradigmcited by 56
  31. 2023Red Teaming Language Model Detectors with Language Modelscited by 55
  32. 2024Signed-Prompt: A New Approach to Prevent Prompt Injection Attacks Against LLM-Integrated Applicationscited by 42
  33. 2023A Comprehensive Overview of Backdoor Attacks in Large Language Models within Communication Networkscited by 38
  34. 2024A Cross-Language Investigation into Jailbreak Attacks in Large Language Modelscited by 37
  35. 2023NOTABLE: Transferable Backdoor Attacks Against Prompt-based NLP Modelscited by 33
  36. 2023Token-Level Adversarial Prompt Detection Based on Perplexity Measures and Contextual Informationcited by 32
  37. 2023Query-Efficient Black-Box Red Teaming via Bayesian Optimizationcited by 31
  38. 2023Model Leeching: An Extraction Attack Targeting LLMscited by 22

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year

Citing papers per year: 2024: 2; 2025: 3.

5 papers.

  1. 2024Attacking Vision-Language Computer Agents via Pop-upscited by 112
  2. 2024Cognitive Overload Attack:Prompt Injection for Long Contextcited by 13
  3. 2025Helping Large Language Models Protect Themselves: An Enhanced Filtering and Summarization Systemcited by 2
  4. 2025Self-reflecting Large Language Models: A Hegelian Dialectical Approachcited by 1
  5. 2025The Laminar Flow Hypothesis: Detecting Jailbreaks via Semantic Turbulence in Large Language Modelscited by 0