# TinyServe: Query-Aware Cache Selection for Efficient LLM Serving

arXiv: https://arxiv.org/abs/2509.12211 · HTML: https://mlsanity.com/paper/2509.12211/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving.
- Builds on 18 of its 33 references that are in the map.
- First posted on arXiv 28 Aug 2025.

## Placed in

- Efficient LLM inference and serving / [KV cache optimization](https://mlsanity.com/inference/kv-cache-optimization/)

## Builds on (most cited first, 18 of 18)

- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Efficient Streaming Language Models with Attention Sinks](https://mlsanity.com/paper/2309.17453/) (2023)
- [LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding](https://mlsanity.com/paper/2308.14508/) (2023)
- [The LAMBADA dataset: Word prediction requiring a broad discourse context](https://mlsanity.com/paper/1606.06031/) (2016)
- [In-context Learning and Induction Heads](https://mlsanity.com/paper/2209.11895/) (2022)
- [Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions](https://mlsanity.com/paper/2212.10509/) (2022)
- [SnapKV: LLM Knows What You are Looking for Before Generation](https://mlsanity.com/paper/2404.14469/) (2024)
- [Progress measures for grokking via mechanistic interpretability](https://mlsanity.com/paper/2301.05217/) (2023)
- [Compressive Transformers for Long-Range Sequence Modelling](https://mlsanity.com/paper/1911.05507/) (2019)
- [TinyLlama: An Open-Source Small Language Model](https://mlsanity.com/paper/2401.02385/) (2024)
- [TinyStories: How Small Can Language Models Be and Still Speak Coherent English?](https://mlsanity.com/paper/2305.07759/) (2023)
- [PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling](https://mlsanity.com/paper/2406.02069/) (2024)
- [FastCache: Fast Caching for Diffusion Transformer Through Learnable Linear Approximation](https://mlsanity.com/paper/2505.20353/) (2025)
- [MT2ST: Adaptive Multi-Task to Single-Task Learning](https://mlsanity.com/paper/2406.18038/) (2024)
- [LLMEasyQuant: Scalable Quantization for Parallel and Distributed LLM Inference](https://mlsanity.com/paper/2406.19657/) (2024)
- [PiKV: KV Cache Management System for Mixture of Experts](https://mlsanity.com/paper/2508.06526/) (2025)

## Built on by (most cited first, 9 of 9)

- [CXL-SpecKV: A Disaggregated FPGA Speculative KV-Cache for Datacenter LLM Serving](https://mlsanity.com/paper/2512.11920/) (2025)
- [CSV-Decode: Certifiable Sub-Vocabulary Decoding for Efficient Large Language Model Inference](https://mlsanity.com/paper/2511.21702/) (2025)
- [AdaCorrection: Adaptive Offset Cache Correction for Accurate Diffusion Transformers](https://mlsanity.com/paper/2602.13357/) (2026)
- [Accelerating Frequency Domain Diffusion Models with Error-Feedback Event-Driven Caching](https://mlsanity.com/paper/2604.22901/) (2026)
- [Thoughts-as-Planning: Latent World Models for Chain-of-Thoughts Optimization via Reinforcement Planning](https://mlsanity.com/paper/2605.28842/) (2026)
- [Cognitive Load Traces as Symbolic and Visual Accounts of Deep Model Cognition](https://mlsanity.com/paper/2510.15980/) (2025)
- [Networking-Aware Energy Efficiency in Agentic AI Inference: A Survey](https://mlsanity.com/paper/2604.07857/) (2026)
- [HSMLA: Hierarchical Softmax Multi-scale Linear Attention for Efficient Vision Transformers](https://mlsanity.com/paper/2608.07616/) (2026)
- [MeshKV: A Network-on-Chip KV Cache Fabric for Scalable Transformer Decoding Accelerators](https://mlsanity.com/paper/2609.19207/) (2026)

## Abstract (arXiv)

Serving large language models (LLMs) efficiently remains challenging due to the high memory and latency overhead of key-value (KV) cache access during autoregressive decoding. We present TinyServe, a lightweight and extensible serving system for deploying tiny LLMs (e.g., TinyLLaMA, GPT2-345M) with support for structured KV sparsity, plugin-based token selection, and hardware-efficient attention kernels. Unlike prior simulation frameworks, TinyServe executes real-time decoding with configurable sparsity strategies and fine-grained instrumentation. To reduce decoding cost, we introduce a textitquery-aware page selection mechanism that leverages bounding-box metadata to estimate attention relevance between the query and KV cache blocks. This enables selective KV loading with minimal overhead and no model modifications. Our fused CUDA kernel integrates page scoring, sparse memory access, and masked attention in a single pass. Experiments show that TinyServe achieves up to 3.4x speedup and over 2x memory savings with negligible accuracy drop. Additional analysis of cache reuse, page hit rate, and multi-GPU scaling confirms its practicality as an efficient system-level design for LLM training and inference research on resource-constrained hardware.