# Exploring Potential Prompt Injection Attacks in Federated Military LLMs and Their Mitigation

arXiv: https://arxiv.org/abs/2501.18416 · HTML: https://mlsanity.com/paper/2501.18416/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security.
- Builds on 17 of its 29 references that are in the map.
- First posted on arXiv 30 Jan 2025.

## Placed in

- LLM safety and security / [Prompt injection and agent security](https://mlsanity.com/safety/prompt-injection-and-agent-security/)

## Builds on (most cited first, 17 of 17)

- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019)
- [Federated Learning: Challenges, Methods, and Future Directions](https://mlsanity.com/paper/1908.07873/) (2019)
- [Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection](https://mlsanity.com/paper/2302.12173/) (2023)
- [Federated Learning with Differential Privacy: Algorithms and Performance Analysis](https://mlsanity.com/paper/1911.00222/) (2019)
- [A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly](https://mlsanity.com/paper/2312.02003/) (2023)
- [Security and Privacy Challenges of Large Language Models: A Survey](https://mlsanity.com/paper/2402.00888/) (2024)
- [On Protecting the Data Privacy of Large Language Models (LLMs): A Survey](https://mlsanity.com/paper/2403.05156/) (2024)
- ["Real Attackers Don't Compute Gradients": Bridging the Gap Between Adversarial ML Research and Practice](https://mlsanity.com/paper/2212.14315/) (2022)
- [Universal Vulnerabilities in Large Language Models: Backdoor Attacks for In-context Learning](https://mlsanity.com/paper/2401.05949/) (2024)
- [PoisonPrompt: Backdoor Attack on Prompt-based Large Language Models](https://mlsanity.com/paper/2310.12439/) (2023)
- [Jailbreaking and Mitigation of Vulnerabilities in Large Language Models](https://mlsanity.com/paper/2410.15236/) (2024)
- [The Future of Large Language Model Pre-training is Federated](https://mlsanity.com/paper/2405.10853/) (2024)
- [Federated Learning Challenges and Opportunities: An Outlook](https://mlsanity.com/paper/2202.00807/) (2022)
- [Emerging Safety Attack and Defense in Federated Instruction Tuning of Large Language Models](https://mlsanity.com/paper/2406.10630/) (2024)
- [Neuro-Symbolic AI for Military Applications](https://mlsanity.com/paper/2408.09224/) (2024)
- [Worldwide Federated Training of Language Models](https://mlsanity.com/paper/2405.14446/) (2024)

## Built on by (most cited first, 5 of 5)

- [A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deployment](https://mlsanity.com/paper/2504.15585/) (2025)
- [Prompt Injection Detection and Mitigation via AI Multi-Agent NLP Frameworks](https://mlsanity.com/paper/2503.11517/) (2025)
- [LLM in the Middle: A Systematic Review of Threats and Mitigations to Real-World LLM-based Systems](https://mlsanity.com/paper/2509.10682/) (2025)
- [CourtGuard: A Local, Multiagent Prompt Injection Classifier](https://mlsanity.com/paper/2510.19844/) (2025)
- [If you're waiting for a sign... that might not be it! Mitigating Trust Boundary Confusion from Visual Injections on Vision-Language Agentic Systems](https://mlsanity.com/paper/2604.19844/) (2026)

## Abstract (arXiv)

Federated Learning (FL) is increasingly being adopted in military collaborations to develop Large Language Models (LLMs) while preserving data sovereignty. However, prompt injection attacks-malicious manipulations of input prompts-pose new threats that may undermine operational security, disrupt decision-making, and erode trust among allies. This perspective paper highlights four vulnerabilities in federated military LLMs: secret data leakage, free-rider exploitation, system disruption, and misinformation spread. To address these risks, we propose a human-AI collaborative framework with both technical and policy countermeasures. On the technical side, our framework uses red/blue team wargaming and quality assurance to detect and mitigate adversarial behaviors of shared LLM weights. On the policy side, it promotes joint AI-human policy development and verification of security protocols.