mlsanityData to 14 Oct 2026

Implicit Manipulation for Skill Selection in LLM Agents with Semantic Matching

arXiv 2609.02035 ↗First posted 2 Sep 2026cs.CRcited by 0

On the map: LLM agents and tool use → Agent skill learning and evolution

Abstract (arXiv)

Skill selection is a key stage in LLM-agent workflows, determining which installed skill should handle a user request. Existing attacks on this stage primarily rely on explicit prompt injection or instruction-level steering, which can expose recognizable manipulation signals. In this work, we identify a new implicit attack surface for skill selection: even when the user prompt and skill description appear benign in isolation, their semantic relationship can still be strategically shaped to favor an attacker-chosen skill. Based on this observation, we present Implicit Skill-Selection Manipulation via Semantic Matching (ISM), which jointly shapes target-skill metadata and reusable prompts to manipulate skill selection without explicit selection instructions. Specifically, we develop a three-stage strategy to broaden semantic coverage, strengthen target distinctiveness, and preserve natural prompt wording. Across four task domains and eight selector models, ISM increases the average target-selection rate (TSR) from 15.2% to 63.5%. In a matched comparison, ISM achieves a 73.5% TSR, only 9.8 percentage points below Explicit Steering. Human reviewers block ISM in only 2.9% of judgments, versus 91.4% for Explicit Steering, while five LLM-based inspectors pass ISM at an average rate of 82.9%, versus 37.4% for Explicit Steering. Moreover, ISM remains effective against PPL-W, Llama Prompt Guard 2, and PIGuard.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Agent skill learning and evolution, one of 16 directions in LLM agents and tool use. Builds on 47 of its 72 references that are in the map. First posted on arXiv 2 Sep 2026.

Builds on

47 of 72 references.

References · 47

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  4. 2025Qwen3 Technical Reportcited by 8,820
  5. 2024GPT-4o System Cardcited by 4,588
  6. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  7. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
  8. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,332
  9. 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
  10. 2024Qwen2.5-Coder Technical Reportcited by 1,203
  11. 2016ConceptNet 5.5: An Open Multilingual Graph of General Knowledgecited by 1,067
  12. 2025OpenAI GPT-5 System Cardcited by 989
35 more
  1. 2021A Survey of Uncertainty in Deep Neural Networkscited by 680
  2. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 651
  3. 2024InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agentscited by 538
  4. 2024StruQ: Defending Against Prompt Injection with Structured Queriescited by 375
  5. 2024Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agentscited by 369
  6. 2023Formalizing and Benchmarking Prompt Injection Attacks and Defensescited by 364
  7. 2025Defeating Prompt Injections by Designcited by 297
  8. 2023MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Usecited by 217
  9. 2025Prompt Injection Attack to Tool Selection in LLM Agentscited by 158
  10. 2024EASYTOOL: Enhancing LLM-based Agents with Concise Tool Instructioncited by 154
  11. 2024AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agentscited by 145
  12. 2024Optimization-based Prompt Injection Attack to LLM-as-a-Judgecited by 143
  13. 2025DataSentinel: A Game-Theoretic Detection of Prompt Injection Attackscited by 143
  14. 2024IsolateGPT: An Execution Isolation Architecture for LLM-Based Agentic Systemscited by 130
  15. 2024The Task Shield: Enforcing Task Alignment to Defend Against Indirect Prompt Injection in LLM Agentscited by 99
  16. 2024ToolSword: Unveiling Safety Issues of Large Language Models in Tool Learning Across Three Stagescited by 82
  17. 2023Tool Documentation Enables Zero-Shot Tool-Usage with Large Language Modelscited by 68
  18. 2023RewriteLM: An Instruction-Tuned Large Language Model for Text Rewritingcited by 67
  19. 2025IPIGuard: A Novel Tool Dependency Graph-Based Defense Against Indirect Prompt Injection in LLM Agentscited by 54
  20. 2024Re-Invoke: Tool Invocation Rewriting for Zero-Shot Tool Retrievalcited by 52
  21. 2025Can Indirect Prompt Injection Attacks Be Detected and Removed?cited by 52
  22. 2021Understanding Hard Negatives in Noise Contrastive Estimationcited by 39
  23. 2025A Critical Evaluation of Defenses against Prompt Injection Attackscited by 36
  24. 2025Defending against Indirect Prompt Injection by Instruction Detectioncited by 32
  25. 2025ToolTweak: An Attack on Tool Selection in LLM-based Agentscited by 26
  26. 2025Attractive Metadata Attack: Inducing LLM Agents to Invoke Malicious Toolscited by 24
  27. 2025ToolScope: Enhancing LLM Agent Tool Use through Tool Merging and Context-Aware Filteringcited by 19
  28. 2021Importance Estimation from Multiple Perspectives for Keyphrase Extractioncited by 16
  29. 2024LLM Whisperer: An Inconspicuous Attack to Bias LLM Responsescited by 15
  30. 2025BiasBusters: Uncovering and Mitigating Tool Selection Bias in Large Language Modelscited by 11
  31. 2025ToolDreamer: Instilling LLM Reasoning Into Tool Retrieverscited by 7
  32. 2024Tailoring Education with GenAI: A New Horizon in Lesson Planningcited by 5
  33. 2025More Than a Score: Probing the Impact of Prompt Specificity on LLM Code Generationcited by 4
  34. 2026Measuring Pragmatic Influence in Large Language Model Instructionscited by 4
  35. 2024Linguistically-Controlled Paraphrase Generationcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Don't Offer What Can't Be Done: Deterministic Executability Gating for LLM Skill Selection at Scale
  2. 2026Convergent Detour Hijacking: Task-Preserving Resource Amplification in Skill-Based LLM Agents
  3. 2026Lost in Implicit Routing: How LLM Agents Fail to Load Relevant Skills at Scale
  4. 2026SkillRet: A Large-Scale Benchmark for Skill Retrieval in LLM Agents
  5. 2026Pretext: Defeating Malicious Skill Detection Frameworks for AI Agents
  6. 2026Optimal Skill Selection for LLM Agents with Provable Bicriteria Guarantees
  7. 2026LatentSkill: From In-Context Textual Skills to In-Weight Latent Skills for LLM Agents
  8. 2026SkillMutator: Benchmarking and Defending Language-and-Code Cross-modal Attacks on LLM Agent Skills