Local-Splitter: A Measurement Study of Seven Tactics for Reducing Cloud LLM Token Usage on Coding-Agent Workloads
On the map: LLM agents and tool use → Agent memory systems
Abstract (arXiv)
We present a systematic measurement study of seven tactics for reducing cloud LLM token usage when a small local model can act as a triage layer in front of a frontier cloud model. The tactics are: (1) local routing, (2) prompt compression, (3) semantic caching, (4) local drafting with cloud review, (5) minimal-diff edits, (6) structured intent extraction, and (7) batching with vendor prompt caching. We implement all seven in an open-source shim that speaks both MCP and the OpenAI-compatible HTTP surface, supporting any local model via Ollama and any cloud model via an OpenAI-compatible endpoint. We evaluate each tactic individually, in pairs, and in a greedy-additive subset across four coding-agent workload classes (edit-heavy, explanation-heavy, general chat, RAG-heavy). We measure tokens saved, dollar cost, latency, and routing accuracy. Our headline finding is that T1 (local routing) combined with T2 (prompt compression) achieves 45-79% cloud token savings on edit-heavy and explanation-heavy workloads, while on RAG-heavy workloads the full tactic set including T4 (draft-review) achieves 51% savings. We observe that the optimal tactic subset is workload-dependent, which we believe is the most actionable finding for practitioners deploying coding agents today.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Agent memory systems, one of 16 directions in LLM agents and tool use. Builds on 16 of its 26 references that are in the map. First posted on arXiv 14 Apr 2026.
Builds on
16 of 26 references.
References · 16
- 2024The Llama 3 Herd of Modelscited by 14,192
- 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
- 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
- 2024Qwen2.5 Technical Reportcited by 4,713
- 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
- 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
- 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
- 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
- 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 821
- 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 791
- 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 633
4 more
- 2024LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compressioncited by 343
- 2023Efficient Guided Generation for Large Language Modelscited by 192
- 2024Large Language Model-Based Agents for Software Engineering: A Surveycited by 173
- 2024Small Language Models: Survey, Measurements, and Insightscited by 109
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026TokenRouter: Efficient Serving System for Token-Level LLM Routing
- 2026Inference Economics of Enterprise Coding Agents: A Case Study of Cloud vs. On-Premise LLMs
- 2026Compact-Memory LLM Agents via Online Max-Member Clustering and Atom-Aware Packing
- 2026Cross-Lingual Token Arbitrage: Optimizing Code Agent Context Windows via Local LLM Preprocessing
- 2025Collaborative Inference and Learning between Edge SLMs and Cloud LLMs: A Survey of Algorithms, Execution, and Open Challenges
- 2025Minions: Cost-efficient Collaboration Between On-device and Cloud Language Models
- 2026Beyond Token Savings: A Systematic Study of Context Compression in LLM Agents
- 2025SageServe: Optimizing LLM Serving on Cloud Data Centers with Forecast Aware Auto-Scaling