On Device Agentic Operation Caches -- Classifier-Centric NL-to-Action Generation
On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks also fits: Agent threats, guardrails and governance
Abstract (arXiv)
Agentic AI is increasingly being embedded in software applications to provide natural language interfaces to features and functionality. In most cases these agents are powered by enterprise (100+ billion parameter) or frontier class large language models that require substantial computational resources run and depend on cloud hosted inference to handle the task of transforming natural language inputs into actionable software operations. This reliance on cloud-hosted inference introduces substantial network latency on top of LLM inference times, creates data privacy concerns, and, given the costs of running these models, can rapidly escalate expenses associated with supporting agentic features. This paper introduces a novel means of converting the NL-to-Action problem from a generative one into a classification-centric formulation via on-device operation caches. These caches allow an agentic system to handle frequently occurring classes of actions completely on-device -- reducing latency, enhancing privacy, and lowering operational costs. We show that for a classic NL-to-Formula task, generating Excel Formula in response to user requests, this approach reduces total inference cost by 56% when compared to cloud-only model-routing based inference and, on cache hits, reduces the latency to response latency by 5x.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 6 of its 16 references that are in the map. First posted on arXiv 27 Sep 2026.
Builds on
6 of 16 references.
References · 6
- 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 853
- 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 682
- 2014Mining Idioms from Source Codecited by 19
- 2025vCache: Verified Semantic Prompt Cachingcited by 17
- 2024MeanCache: User-Centric Semantic Caching for LLM Web Servicescited by 14
- 2025Generative Caching for Structurally Similar Prompts and Responsescited by 3
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025AgentFlux: Decoupled Fine-Tuning & Inference for On-Device Agentic Systems
- 2026Why Agent Caching Fails and How to Fix It: Structured Intent Canonicalization with Few-Shot Learning
- 2025Optimizing Agentic Language Model Inference via Speculative Tool Calls
- 2026ThunderAgent: A Simple, Fast and Program-Aware Agentic Inference System
- 2026MemDecay: Region-Aware KV Cache Eviction for Efficient LLM Agent Inference
- 2025Agentic Privacy-Preserving Machine Learning
- 2026Agent libOS: A Runtime Substrate for Capability-Controlled Self-Evolving LLM Agents
- 2026Efficient LLM Serving for Agentic Workflows: A Data Systems Perspective