# Extracting Knowledge from Tools in LLM Agents

arXiv: https://arxiv.org/abs/2608.30288 · HTML: https://mlsanity.com/paper/2608.30288/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Tool learning and function calling, one of 16 directions in LLM agents and tool use.
- Builds on 14 of its 51 references that are in the map.
- First posted on arXiv 31 Aug 2026.

## Placed in

- LLM agents and tool use / [Tool learning and function calling](https://mlsanity.com/agents/tool-learning-and-function-calling/)

## Builds on (most cited first, 14 of 14)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [The Landscape of Emerging AI Agent Architectures for Reasoning, Planning, and Tool Calling: A Survey](https://mlsanity.com/paper/2404.11584/) (2024)
- [Follow My Instruction and Spill the Beans: Scalable Data Extraction from Retrieval-Augmented Generation Systems](https://mlsanity.com/paper/2402.17840/) (2024)
- [DeepAgent: A General Reasoning Agent with Scalable Toolsets](https://mlsanity.com/paper/2510.21618/) (2025)
- [LLM Agents for Education: Advances and Applications](https://mlsanity.com/paper/2503.11733/) (2025)
- [ToolTalk: Evaluating Tool-Usage in a Conversational Setting](https://mlsanity.com/paper/2311.10775/) (2023)
- [Feedback-Guided Extraction of Knowledge Base from Retrieval-Augmented LLM Applications](https://mlsanity.com/paper/2411.14110/) (2024)
- [Unleashing Worms and Extracting Data: Escalating the Outcome of Attacks against RAG-based Inference in Scale and Severity Using Jailbreaking](https://mlsanity.com/paper/2409.08045/) (2024)
- [RAG-MCP: Mitigating Prompt Bloat in LLM Tool Selection via Retrieval-Augmented Generation](https://mlsanity.com/paper/2505.03275/) (2025)
- [Silent Leaks: Implicit Knowledge Extraction Attack on RAG Systems through Benign Queries](https://mlsanity.com/paper/2505.15420/) (2025)
- [WTU-EVAL: A Whether-or-Not Tool Usage Evaluation Benchmark for Large Language Models](https://mlsanity.com/paper/2407.12823/) (2024)
- [ADAM: A Systematic Data Extraction Attack on Agent Memory via Adaptive Querying](https://mlsanity.com/paper/2604.09747/) (2026)
- [Connect the Dots: Knowledge Graph-Guided Crawler Attack on Retrieval-Augmented Generation Systems](https://mlsanity.com/paper/2601.15678/) (2026)
- [Isolated but Exposed: Persistence-Based Memory Extraction Attack on LLM Agents](https://mlsanity.com/paper/2607.23444/) (2026)

## Abstract (arXiv)

LLM agents commonly use knowledge-based tools and access their underlying files, databases, and search indexes through tool invocation. This integration improves agents' ability to provide domain-specific services but also introduces the risk of tool-mediated knowledge extraction: source content exposed to an agent for legitimate responses may be progressively recovered from its outputs, enabling reconstruction of the knowledge source behind a target tool. This paper systematically investigates this risk and identifies two challenges introduced by tool invocation: tool-selection uncertainty, where an agent may invoke a competing tool instead of the target tool, and tool-argument compression, where fine-grained query information may be lost when the agent generates tool arguments. To tackle these challenges, we propose ToolSiphon, a query-only extraction attack that introduces two complementary signals: a target-discriminative signal, implemented through Tool Contrastive Analysis, to steer queries toward the target tool; and a response-grounded factual signal, implemented through Evidence Chained Feedback, to mitigate argument compression and progressively expand extraction coverage. Across three types of knowledge-based tools and six domain-specific datasets, ToolSiphon recovers 74.3% of source records on average when coarse-grained information about non-target tools is available, with 83.2% textual recovery and 90.2% semantic similarity. Even without such information, it recovers 66.3% of source records. ToolSiphon also remains effective against representative defenses and on three real-world agent platforms.