# LLM-dCache: Improving Tool-Augmented LLMs with GPT-Driven Localized Data Caching

arXiv: https://arxiv.org/abs/2406.06799 · HTML: https://mlsanity.com/paper/2406.06799/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 20 of its 26 references that are in the map.
- First posted on arXiv 10 Jun 2024.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 20 of 20)

- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023)
- [ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs](https://mlsanity.com/paper/2307.16789/) (2023)
- [MemGPT: Towards LLMs as Operating Systems](https://mlsanity.com/paper/2310.08560/) (2023)
- [LLM-Pruner: On the Structural Pruning of Large Language Models](https://mlsanity.com/paper/2305.11627/) (2023)
- [ToolQA: A Dataset for LLM Question Answering with External Tools](https://mlsanity.com/paper/2306.13304/) (2023)
- [RouterBench: A Benchmark for Multi-LLM Routing System](https://mlsanity.com/paper/2403.12031/) (2024)
- [An LLM Compiler for Parallel Function Calling](https://mlsanity.com/paper/2312.04511/) (2023)
- [LLM Inference Unveiled: Survey and Roofline Model Insights](https://mlsanity.com/paper/2402.16363/) (2024)
- [Single-Path NAS: Designing Hardware-Efficient ConvNets in less than 4 Hours](https://mlsanity.com/paper/1904.02877/) (2019)
- [GeoLLM-Engine: A Realistic Environment for Building Geospatial Copilots](https://mlsanity.com/paper/2404.15500/) (2024)
- [Evaluating Tool-Augmented Agents in Remote Sensing Platforms](https://mlsanity.com/paper/2405.00709/) (2024)
- [Cached Model-as-a-Resource: Provisioning Large Language Model Agents for Edge Intelligence in Space-air-ground Integrated Networks](https://mlsanity.com/paper/2403.05826/) (2024)
- [An LLM-Tool Compiler for Fused Parallel Function Calling](https://mlsanity.com/paper/2405.17438/) (2024)
- [Retrieval-augmented GPT-3.5-based Text-to-SQL Framework with Sample-aware Prompting and Dynamic Revision Chain](https://mlsanity.com/paper/2307.05074/) (2023)
- [GeckOpt: LLM System Efficiency via Intent-Based Tool Selection](https://mlsanity.com/paper/2404.15804/) (2024)
- [Hardware-Aware Machine Learning: Modeling and Optimization](https://mlsanity.com/paper/1809.05476/) (2018)
- [Evaluating Zero-Shot GPT-4V Performance on 3D Visual Question Answering Benchmarks](https://mlsanity.com/paper/2405.18831/) (2024)
- [Unlearning Climate Misinformation in Large Language Models](https://mlsanity.com/paper/2405.19563/) (2024)

## Built on by (most cited first, 7 of 7)

- [Asynchronous LLM Function Calling](https://mlsanity.com/paper/2412.07017/) (2024)
- [Automated Multi-Agent Workflows for RTL Design](https://mlsanity.com/paper/2509.20182/) (2025)
- [Generative Caching for Structurally Similar Prompts and Responses](https://mlsanity.com/paper/2511.17565/) (2025)
- [Less is More: Optimizing Function Calling for LLM Execution on Edge Devices](https://mlsanity.com/paper/2411.15399/) (2024)
- [CarbonCall: Sustainability-Aware Function Calling for Large Language Models on Edge Devices](https://mlsanity.com/paper/2504.20348/) (2025)
- [Foundation-Model-Based Agents in Industrial Automation: Purposes, Capabilities, and Open Challenges](https://mlsanity.com/paper/2605.02592/) (2026)
- [MultiFluxAI Enhancing Platform Engineering with Advanced Agent-Orchestrated Retrieval Systems](https://mlsanity.com/paper/2508.21307/) (2025)

## Abstract (arXiv)

As Large Language Models (LLMs) broaden their capabilities to manage thousands of API calls, they are confronted with complex data operations across vast datasets with significant overhead to the underlying system. In this work, we introduce LLM-dCache to optimize data accesses by treating cache operations as callable API functions exposed to the tool-augmented agent. We grant LLMs the autonomy to manage cache decisions via prompting, seamlessly integrating with existing function-calling mechanisms. Tested on an industry-scale massively parallel platform that spans hundreds of GPT endpoints and terabytes of imagery, our method improves Copilot times by an average of 1.24x across various LLMs and prompting techniques.