# SEEM: Exploiting Black-Box Text Attacks to Manipulate Tool Selection

arXiv: https://arxiv.org/abs/2504.04809 · HTML: https://mlsanity.com/paper/2504.04809/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Tool learning and function calling, one of 16 directions in LLM agents and tool use.
- First posted on arXiv 7 Apr 2025.

## Placed in

- LLM agents and tool use / [Tool learning and function calling](https://mlsanity.com/agents/tool-learning-and-function-calling/)

## Built on by (most cited first, 2 of 2)

- [RRTL: Red Teaming Reasoning Large Language Models in Tool Learning](https://mlsanity.com/paper/2505.17106/) (2025)
- [ToolFlood: Beyond Selection -- Hiding Valid Tools from LLM Agents via Semantic Covering](https://mlsanity.com/paper/2603.13950/) (2026)

## Abstract (arXiv)

Tool learning has emerged as a powerful auxiliary mechanism that extends the capabilities of large language models (LLMs), enabling them to address complex tasks that demand real-time relevance or high-precision operations. However, beneath this strength lie significant security risks. Prior studies have primarily concentrated on corrupting the outputs of invoked tools, while largely overlooking the vulnerability of the tool selection process itself. To bridge this gap, we introduce a black-box, text-based attack that substantially increases the likelihood of a target tool being selected. We propose SEEM, a two-level coarse-to-fine perturbation method that operates at both the word and character levels. Through comprehensive experiments, we show that merely perturbing the textual information of tools can markedly raise the probability of the target tool being prioritized and ranked higher among candidates. Our findings expose critical weaknesses in the tool selection mechanism and lay the groundwork for developing defenses to secure this essential process.