Harmless Yet Harmful: Neutral Prompting Attacks for Stealthy Hallucination Steering in Agent Skills
On the map: LLM agents and tool use → Agent threats, guardrails and governance
Abstract (arXiv)
LLM-powered coding agents increasingly participate in software development workflows by generating code, selecting dependencies, and producing package installation commands. This creates a new software supply chain risk: when an agent hallucinates a non-existent package, an attacker may register the hallucinated name and later compromise users who install it. Existing package hallucination attacks and defenses primarily focus on naturally occurring hallucinations, targeted dependency steering, or post-hoc package validation. In this paper, we introduce Neutral Prompting Attack (NPA), a highly stealthy attack paradigm in which semantically benign instructions, such as encouraging imagination and exhaustiveness, increase package hallucination propensity without containing explicit malicious intent. Unlike targeted dependency steering, NPA does not specify an attacker-chosen package. Instead, it shifts the model's dependency generation behavior toward more speculative package names. We evaluate NPA across multiple coding-oriented LLMs and package hallucination benchmarks. Our results show that NPA increases both Hallucination ASR and Pip Install ASR, changes the distribution of hallucinated package names, and evades existing static-analysis, LLM-based, and agent-based Skill defenses. These findings reveal that harmless-looking prompts can covertly manipulate hallucination behavior and create downstream software supply chain risks.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 11 of its 18 references that are in the map. First posted on arXiv 28 May 2026.
Builds on
11 of 18 references.
Closest ancestors · 1
Same direction, newest first
Other references · 10
- 2025Why Language Models Hallucinatecited by 220
- 2023LLM Lies: Hallucinations are not Bugs, but Features as Adversarial Examplescited by 177
- 2024We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMscited by 65
- 2024A Survey on Large Language Model Hallucination via a Creativity Perspectivecited by 38
- 2025The Illusion of Progress: Re-evaluating Hallucination Detection in LLMscited by 22
- 2024RoleBreak: Character Hallucination as a Jailbreak Attack in Role-Playing Systemscited by 21
- 2025Importing Phantoms: Measuring LLM Package Hallucination Vulnerabilitiescited by 15
- 2025HalluShift: Measuring Distribution Shifts towards Hallucination Detection in LLMscited by 8
- 2026Steering Language Models Before They Speak: Logit-Level Interventionscited by 4
- 2025The Illusionist's Prompt: Exposing the Factual Vulnerabilities of Large Language Models with Linguistic Nuancescited by 2
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026Package Hallucination Attacks on Coding Agents through Prompt Injection in Rule Files
- 2026LLM Ghostbusters: Surgical Package Hallucination Suppression via Adaptive Unlearning
- 2026Evaluating Inference-Time Defenses Against Package Hallucination in LLM-Generated Code
- 2026Closed-World Resolution Against Tool Hallucination in LLM Agents
- 2026PhantomSkill: Malicious Code Injection in Agent Skill Ecosystems
- 2026Understanding and Mitigating Hallucination Escape in Tool-Using LLM Agents
- 2026Skills That Don't Exist: A Large-Scale Study of Hallucinated Skill Recommendation in LLM Agents
- 2026Refusing the Impossible: A Taxonomy and Benchmark for Code Hallucination in Large Language Models ↗