# FIER: Fine-Grained and Efficient KV Cache Retrieval for Long-context LLM Inference

arXiv: https://arxiv.org/abs/2508.08256 · HTML: https://mlsanity.com/paper/2508.08256/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 May 2025.

## Built on by (most cited first, 3 of 3)

- [MuKV: Multi-Grained KV Cache Compression for Long Streaming Video Question-Answering](https://mlsanity.com/paper/2605.22269/) (2026)
- [A Guide to Large Language Models in Modeling and Simulation: From Core Techniques to Critical Challenges](https://mlsanity.com/paper/2602.05883/) (2026)
- [TTKV: Temporal-Tiered KV Cache for Long-Context LLM Inference](https://mlsanity.com/paper/2604.19769/) (2026)

## Abstract (arXiv)

The Key-Value (KV) cache reading latency increases significantly with context lengths, hindering the efficiency of long-context LLM inference. To address this, previous works propose retaining a small fraction of KV cache based on token importance. For example, KV eviction uses static heuristics to retain tokens, while KV retrieval dynamically selects query-relevant tokens for more adaptive cache management. However, we observe that important tokens are often sparsely distributed across the long context. This sparsity makes existing page-level KV retrieval inaccurate, as each page may include irrelevant tokens and miss critical ones. In this work, we propose Fier, a underlineFine-Grained and underlineEfficient KV cache underlineRetrieval method. Fier uses 1-bit quantized keys to estimate the importance of each token, resulting in efficient and precise retrieval. Experiments show that Fier matches full KV performance using only 11% of the cache budget across various long-context tasks, reducing decoding latency by 1.2× to 1.5×.Code is available at https://github.com/SimWangArizona/FIER