Inference Economics of Enterprise Coding Agents: A Case Study of Cloud vs. On-Premise LLMs
On the map: LLM agents and tool use → Agent memory systems
Abstract (arXiv)
Autonomous coding agents force engineering organizations to choose between API-based frontier models -- strong reasoning at high token cost -- and on-premise quantized open-weights models, which promise low-marginal-cost scaling and data sovereignty at some loss of reasoning fidelity. We study this trade-off through a single-developer, non-randomized longitudinal case study over two contiguous 28-day periods on a production monorepo: an API-based Claude Opus 4.7/4.8 configuration using Claude Code versus an on-premise GLM-5.1/5.2 configuration using Opencode, quantized to NVFP4, on NVIDIA Blackwell hardware. Analyzing LLM telemetry and Git history, we find that prompt caching (99.3% hit rate) cuts realized API cost by 88.6% to an effective 0.57 per million tokens -- below even the 2.83 amortized unit cost of the shared on-premise slice (a utilization-dependent inversion; total realized spend and total cost of ownership (TCO) are the robust quantities). At comparable gross code churn, the local configuration was associated with a far higher defect-repair burden: a Fix Commit Ratio (FCR) of 74.9% versus 45.9%, with the odds of a commit being a repair 2.6 to 4.9 times higher within every difficulty tier (Mantel-Haenszel OR = 3.61). Under Taiwan-market parameters and a symmetric labor model, on-premise deployment nonetheless saves 40.1% of true TCO under shared GPU allocation, whereas dedicated reservation costs 43.8% more than the cached API. Under shared allocation, the genuine penalty is not monetary but a measurable developer-experience burden -- timestamp indicators show more work trapped in debugging spirals and a slower commit cadence -- and an offline replay shows hybrid routing gateways trade defect rate for infrastructure savings along a cost-quality frontier rather than dominate the pure-API baseline.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Agent memory systems, one of 16 directions in LLM agents and tool use. Builds on 16 of its 36 references that are in the map. First posted on arXiv 13 Jul 2026.
Builds on
16 of 36 references.
References · 16
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
- 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
- 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
- 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,737
- 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
- 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
- 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 853
- 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 682
- 2024Agentless: Demystifying LLM-based Software Engineering Agentscited by 421
- 2022CodeT: Code Generation with Generated Testscited by 384
- 2023The Impact of AI on Developer Productivity: Evidence from GitHub Copilotcited by 274
- 2022Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Modelcited by 265
4 more
- 2024AI Agents That Mattercited by 219
- 2025Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivitycited by 101
- 2019On the Impact of Programming Languages on Code Qualitycited by 6
- 2025Evaluating the Impact of Post-Training Quantization on Large Language Models for Code Generationcited by 3
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026The Harness Effect: How Orchestration Design Sets the Token Economics of Enterprise Agentic AI
- 2025The Cost of Dynamic Reasoning: Demystifying AI Agents and Test-Time Scaling from an AI Infrastructure Perspective
- 2026How Do AI Agents Spend Your Money? Analyzing and Predicting Token Consumption in Agentic Coding Tasks
- 2026ISO-Bench: Can Coding Agents Optimize Real-World Inference Workloads?
- 2026LLMoxie: Exploring Agentic AI for Scientific Software Development
- 2026Evaluating Inference Compute for Generative AI: A Framework for Enterprise Workloads
- 2026Local-Splitter: A Measurement Study of Seven Tactics for Reducing Cloud LLM Token Usage on Coding-Agent Workloads
- 2026ABC-Bench: Benchmarking Agentic Backend Coding in Real-World Development