# Cold-RL: Learning Cache Eviction with Offline Reinforcement Learning for NGINX

arXiv: https://arxiv.org/abs/2508.12485 · HTML: https://mlsanity.com/paper/2508.12485/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 1 of its 10 references that are in the map.
- First posted on arXiv 17 Aug 2025.

## Builds on (most cited first, 1 of 1)

- [Competitive caching with machine learned advice](https://mlsanity.com/paper/1802.05399/) (2018)

## Built on by (most cited first, 1 of 1)

- [SCION: Size-aware Policy Orchestration for Nonstationary Object Caches (Long Paper Version)](https://mlsanity.com/paper/2605.01055/) (2026)

## Abstract (arXiv)

Web proxies such as NGINX commonly rely on least-recently-used (LRU) eviction, which is size agnostic and can thrash under periodic bursts and mixed object sizes. We introduce Cold-RL, a learned eviction policy for NGINX that replaces LRU's forced-expire path with a dueling Deep Q-Network served by an ONNX sidecar within a strict microsecond budget. On each eviction, Cold-RL samples the K least-recently-used objects, extracts six lightweight features (age, size, hit count, inter-arrival time, remaining TTL, and last origin RTT), and requests a bitmask of victims; a hard timeout of 500 microseconds triggers immediate fallback to native LRU. Policies are trained offline by replaying NGINX access logs through a cache simulator with a simple reward: a retained object earns one point if it is hit again before TTL expiry. We compare against LRU, LFU, size-based, adaptive LRU, and a hybrid baseline on two adversarial workloads. With a 25 MB cache, Cold-RL raises hit ratio from 0.1436 to 0.3538, a 146 percent improvement over the best classical baseline; at 100 MB, from 0.7530 to 0.8675, a 15 percent gain; and at 400 MB it matches classical methods (about 0.918). Inference adds less than 2 percent CPU overhead and keeps 95th percentile eviction latency within budget. To our knowledge, this is the first reinforcement learning eviction policy integrated into NGINX with strict SLOs.