# Improving Tool Retrieval by Leveraging Large Language Models for Query Generation

arXiv: https://arxiv.org/abs/2412.03573 · HTML: https://mlsanity.com/paper/2412.03573/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 19 references that are in the map.
- First posted on arXiv 17 Nov 2024.

## Builds on (most cited first, 14 of 14)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs](https://mlsanity.com/paper/2307.16789/) (2023)
- [Gorilla: Large Language Model Connected with Massive APIs](https://mlsanity.com/paper/2305.15334/) (2023)
- [Unsupervised Dense Information Retrieval with Contrastive Learning](https://mlsanity.com/paper/2112.09118/) (2021)
- [Learning To Retrieve Prompts for In-Context Learning](https://mlsanity.com/paper/2112.08633/) (2021)
- [Pretrained Transformers for Text Ranking: BERT and Beyond](https://mlsanity.com/paper/2010.06467/) (2020)
- [Unified Demonstration Retriever for In-Context Learning](https://mlsanity.com/paper/2305.04320/) (2023)
- [Are LLMs All You Need for Task-Oriented Dialogue?](https://mlsanity.com/paper/2304.06556/) (2023)
- [Novelty Controlled Paraphrase Generation with Retrieval Augmented Conditional Prompt Tuning](https://mlsanity.com/paper/2202.00535/) (2022)
- [Scalable and Robust Self-Learning for Skill Routing in Large-Scale Conversational AI Systems](https://mlsanity.com/paper/2204.07135/) (2022)

## Built on by (most cited first, 11 of 11)

- [ToolOmni: Enabling Open-World Tool Use via Agentic learning with Proactive Retrieval and Grounded Execution](https://mlsanity.com/paper/2604.13787/) (2026)
- [FinTrace: Holistic Trajectory-Level Evaluation of LLM Tool Calling for Long-Horizon Financial Tasks](https://mlsanity.com/paper/2604.10015/) (2026)
- [MassTool: A Multi-Task Search-Based Tool Retrieval Framework for Large Language Models](https://mlsanity.com/paper/2507.00487/) (2025)
- [Discrepancy Detection at the Data Level: Toward Consistent Multilingual Question Answering](https://mlsanity.com/paper/2510.11928/) (2025)
- [ScaleCall -- Agentic Tool Calling at Scale for Fintech: Challenges, Methods, and Deployment Insights](https://mlsanity.com/paper/2511.00074/) (2025)
- [From Large to Super-Tiny: End-to-End Optimization for Cost-Efficient LLMs](https://mlsanity.com/paper/2504.13471/) (2025)
- [Bidirectional Semantic Complementary Tool Retrieval for Remote Sensing Agents](https://mlsanity.com/paper/2606.07538/) (2026)
- [JSPLIT: A Taxonomy-based Solution for Prompt Bloating in Model Context Protocol](https://mlsanity.com/paper/2510.14537/) (2025)
- [Toollery: Scaling LLM Agents to Thousands of Skills and Tools](https://mlsanity.com/paper/2609.22218/) (2026)
- [ParaAgent: Reinforcing Parallel Acting in Open-World Tool Environments](https://mlsanity.com/paper/2609.33618/) (2026)
- [Self-Evolving Agents via Likelihood-Guided Tool-Space Optimization](https://mlsanity.com/paper/2609.34151/) (2026)

## Abstract (arXiv)

Using tools by Large Language Models (LLMs) is a promising avenue to extend their reach beyond language or conversational settings. The number of tools can scale to thousands as they enable accessing sensory information, fetching updated factual knowledge, or taking actions in the real world. In such settings, in-context learning by providing a short list of relevant tools in the prompt is a viable approach. To retrieve relevant tools, various approaches have been suggested, ranging from simple frequency-based matching to dense embedding-based semantic retrieval. However, such approaches lack the contextual and common-sense understanding required to retrieve the right tools for complex user requests. Rather than increasing the complexity of the retrieval component itself, we propose leveraging LLM understanding to generate a retrieval query. Then, the generated query is embedded and used to find the most relevant tools via a nearest-neighbor search. We investigate three approaches for query generation: zero-shot prompting, supervised fine-tuning on tool descriptions, and alignment learning by iteratively optimizing a reward metric measuring retrieval performance. By conducting extensive experiments on a dataset covering complex and multi-tool scenarios, we show that leveraging LLMs for query generation improves the retrieval for in-domain (seen tools) and out-of-domain (unseen tools) settings.