A Call to Action for a Secure-by-Design Generative AI Paradigm
On the map: LLM safety and security → Prompt injection and agent security
Abstract (arXiv)
Large language models have gained widespread prominence, yet their vulnerability to prompt injection and other adversarial attacks remains a critical concern. This paper argues for a security-by-design AI paradigm that proactively mitigates LLM vulnerabilities while enhancing performance. To achieve this, we introduce PromptShield, an ontology-driven framework that ensures deterministic and secure prompt interactions. It standardizes user inputs through semantic validation, eliminating ambiguity and mitigating adversarial manipulation. To assess PromptShield's security and performance capabilities, we conducted an experiment on an agent-based system to analyze cloud logs within Amazon Web Services (AWS), containing 493 distinct events related to malicious activities and anomalies. By simulating prompt injection attacks and assessing the impact of deploying PromptShield, our results demonstrate a significant improvement in model security and performance, achieving precision, recall, and F1 scores of approximately 94%. Notably, the ontology-based framework not only mitigates adversarial threats but also enhances the overall performance and reliability of the system. Furthermore, PromptShield's modular and adaptable design ensures its applicability beyond cloud security, making it a robust solution for safeguarding generative AI applications across various domains. By laying the groundwork for AI safety standards and informing future policy development, this work stimulates a crucial dialogue on the pivotal role of deterministic prompt engineering and ontology-based validation in ensuring the safe and responsible deployment of LLMs in high-stakes environments.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 21 of its 64 references that are in the map. First posted on arXiv 1 Oct 2025.
Builds on
21 of 64 references.
Closest ancestors · 6
Same direction, newest first
- 2024Prompt Infection: LLM-to-LLM Prompt Injection within Multi-Agent Systems9 Oct
- 2024Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)20 Jul
- 2024Automatic and Universal Prompt Injection Attacks against Large Language Models7 Mar
- 2024A Novel Evaluation Framework for Assessing Resilience Against Prompt Injection Attacks in Large Language Models2 Jan
- 2023Ignore This Title and HackAPrompt: Exposing Systemic Vulnerabilities of LLMs through a Global Scale Prompt Hacking Competition24 Oct
- 2023Prompt Injection attack against LLM-integrated Applications8 Jun
Other references · 15
- 2022Training language models to follow instructions with human feedbackcited by 15,712
- 2014Explaining and Harnessing Adversarial Examplescited by 7,287
- 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
- 2016Towards Evaluating the Robustness of Neural Networkscited by 2,976
- 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
- 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
- 2019Certified Adversarial Robustness via Randomized Smoothingcited by 1,324
- 2022In-context Learning and Induction Headscited by 930
- 2024A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applicationscited by 428
- 2024Flooding Spread of Manipulated Knowledge in LLM-Based Multi-Agent Communitiescited by 84
- 2024"We Need Structured Output": Towards User-centered Constraints on Large Language Model Outputcited by 46
- 2024A Survey of Prompt Engineering Methods in Large Language Models for Different NLP Taskscited by 46
Built on by
Citing papers per year: 2026: 3.
3 papers.