# Saving GPU Hours in LLM Inference System Development and Online Workloads with Simulation and DBMS-Inspired Cache Replacement Policies

arXiv: https://arxiv.org/abs/2411.07447 · HTML: https://mlsanity.com/paper/2411.07447/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 0 of its 10 references that are in the map.
- First posted on arXiv 12 Nov 2024.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)
- Efficient LLM inference and serving / [Serving systems and request scheduling](https://mlsanity.com/inference/serving-systems-and-request-scheduling/)

## Built on by (most cited first, 3 of 3)

- [Echo: Efficient Co-Scheduling of Hybrid Online-Offline Tasks for Large Language Model Serving](https://mlsanity.com/paper/2504.03651/) (2025)
- [Mitigating KV Cache Competition to Enhance User Experience in LLM Inference](https://mlsanity.com/paper/2503.13773/) (2025)
- [Play by the Type Rules: Inferring Constraints for LLM Functions in Declarative Programs](https://mlsanity.com/paper/2509.20208/) (2025)

## Abstract (arXiv)

LLMs are increasingly used world-wide from daily tasks to agentic systems and data analytics, requiring significant GPU resources. While LLM inference systems are capable of serving millions of requests from multiple users, they often lack theoretical models to determine whether they achieve the performance upper bounds of underlying hardware resources. Beyond online workload serving, merely analyzing existing systems-or developing yet another one-is both GPU-intensive and labor-intensive. This paper provides a comprehensive survey of LLM inference systems, focusing on their cache management policies and availability. We then show that simulations can be an effective tool to save GPU hours in the development and analysis phase of inference systems, revealing useful insights for developing better inference techniques, unlike how existing studies used simulations to find the best parameters inside a given system. Finally, we provide theoretical tools to estimate the optimal performance and formulate new ideas. Based on the theoretical analysis, especially on the cache management in LLM inference, we propose a simple yet effective cache replacement policy that can be easily plugged into existing preemptive schedulers and systems. We show that such a simple policy inspired from database systems can substantially save GPU hours in actual inference systems on online workloads. We share our experience submitting a journal paper to a database venue in November 2025 for anyone considering a similar path.