Poison Attacks and Adversarial Prompts Against an Informed University Virtual Assistant
On the map: LLM safety and security → Prompt injection and agent security
Abstract (arXiv)
Recent research has shown that large language models (LLMs) are particularly vulnerable to adversarial attacks. Since the release of ChatGPT, various industries are adopting LLM-based chatbots and virtual assistants in their data workflows. The rapid development pace of AI-based systems is being driven by the potential of Generative AI (GenAI) to assist humans in decision making. The immense optimism behind GenAI often overshadows the adversarial risks associated with these technologies. A threat actor can use security gaps, poor safeguards, and limited data governance to carry out attacks that grant unauthorized access to the system and its data. As a proof-of-concept, we assess the performance of BarkPlug, the Mississippi State University chatbot, against data poison attacks from a red team perspective.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 8 of its 11 references that are in the map. First posted on arXiv 3 Nov 2024.
Builds on
8 of 11 references.
References · 8
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
- 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
- 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
- 2024PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Modelscited by 267
- 2024Typos that Broke the RAG's Back: Genetic Attack on RAG Pipeline by Simulating Documents in the Wild via Low-level Perturbationscited by 65
- 2024Accelerating Greedy Coordinate Gradient and General Prompt Optimization via Probe Samplingcited by 37
- 2024From Questions to Insightful Answers: Building an Informed Chatbot for University Resourcescited by 12
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025A Proposal for Evaluating the Operational Risk for ChatBots based on Large Language Models
- 2023From ChatGPT to ThreatGPT: Impact of Generative AI in Cybersecurity and Privacy
- 2025GenAI Security: Outsmarting the Bots with a Proactive Testing Framework
- 2023Identifying and Mitigating the Security Risks of Generative AI
- 2024An Early Categorization of Prompt Injection Attacks on Large Language Models
- 2024Review of Generative AI Methods in Cybersecurity
- 2025ChatInject: Abusing Chat Templates for Prompt Injection in LLM Agents
- 2026Detection of adversarial intent in Human-AI teams using LLMs