Breaking to Build: A Threat Model of Prompt-Based Attacks for Securing LLMs
On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks
also in 1 more
Abstract (arXiv)
The proliferation of Large Language Models (LLMs) has introduced critical security challenges, where adversarial actors can manipulate input prompts to cause significant harm and circumvent safety alignments. These prompt-based attacks exploit vulnerabilities in a model's design, training, and contextual understanding, leading to intellectual property theft, misinformation generation, and erosion of user trust. A systematic understanding of these attack vectors is the foundational step toward developing robust countermeasures. This paper presents a comprehensive literature survey of prompt-based attack methodologies, categorizing them to provide a clear threat model. By detailing the mechanisms and impacts of these exploits, this survey aims to inform the research community's efforts in building the next generation of secure LLMs that are inherently resistant to unauthorized distillation, fine-tuning, and editing.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 17 of its 27 references that are in the map. First posted on arXiv 4 Sep 2025.
Builds on
17 of 27 references.
Closest ancestors · 1
Same direction, newest first
Other references · 16
- 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
- 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,443
- 2023Prompt Injection attack against LLM-integrated Applicationscited by 605
- 2023Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attackscited by 278
- 2023PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Promptscited by 226
- 2023LLM Lies: Hallucinations are not Bugs, but Features as Adversarial Examplescited by 177
- 2024BadChain: Backdoor Chain-of-Thought Prompting for Large Language Modelscited by 80
- 2023Ignore This Title and HackAPrompt: Exposing Systemic Vulnerabilities of LLMs through a Global Scale Prompt Hacking Competitioncited by 68
- 2023Black Box Adversarial Prompting for Foundation Modelscited by 62
- 2023Tricking LLMs into Disobedience: Formalizing, Analyzing, and Detecting Jailbreakscited by 49
- 2023TrojLLM: A Black-box Trojan Prompt Attack on Large Language Modelscited by 31
- 2023Identifying and Mitigating Privacy Risks Stemming from Language Models: A Surveycited by 30
4 more
- 2023TrojText: Test-time Invisible Textual Trojan Insertioncited by 28
- 2024Trojan Detection in Large Language Models: Insights from The Trojan Detection Challengecited by 14
- 2022A Prompting-based Approach for Adversarial Example Generation and Robustness Enhancementcited by 12
- 2024Privacy Risks of General-Purpose AI Systems: A Foundation for Investigating Practitioner Perspectivescited by 5
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2024Exploring Vulnerabilities and Protections in Large Language Models: A Survey
- 2025Paper Summary Attack: Jailbreaking LLMs through LLM Safety Papers
- 2025Attack and defense techniques in large language models: A survey and new perspectives
- 2024Breaking Down the Defenses: A Comparative Survey of Attacks on Large Language Models
- 2023Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attacks
- 2026Analysis of LLMs Against Prompt Injection and Jailbreak Attacks
- 2026An Empirical Evaluation of Prompt Injection Vulnerabilities in Large Language Models Across Multilingual and Obfuscated Attack Scenarios
- 2025SecurityLingua: Efficient Defense of LLM Jailbreak Attacks via Security-Aware Prompt Compression