mlsanityData to 7 Oct 2026

TierKV: Long-Context On-Device LLMs via Predictive Multi-Tier KV Caching

arXiv 2609.21172 ↗First posted 18 Sep 2026cs.LG cs.SY eess.SYcited by 0

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Large language models (LLMs) are moving onto mobile devices for increasingly diverse workloads over text, images, video, and audio. These applications often require long contexts, making the Key-Value (KV) cache a dominant memory bottleneck because it grows linearly with sequence length and is accessed at every decoding step. Prior work reduces KV-cache footprint through low-rank compression, token eviction, or flash offloading, but the resulting reconstruction overhead, irreversible token loss, or I/O stalls can offset the benefit of saving memory. We present TierKV, a mobile LLM inference framework built on Predictive Multi-Tier Cache Optimization (PMCO). Before decoding starts, PMCO predicts future cache demand from prefill hidden states and jointly assigns tokens to exact, low-rank, and flash-offloaded tiers under the device memory and accuracy budgets. This formulation retains access to the full context, removes the circular dependency of reactive eviction, and admits a closed-form solver that selects tier boundaries and per-layer ranks at runtime. Across eight text, vision, and audio models on three mobile SoCs, TierKV improves prefill throughput by up to 17.6x over existing mobile LLM frameworks, reduces RAM-resident KV cache by 12.5-34%, thereby enabling substantially longer contexts under the same memory budget, while incurring only minor accuracy degradation.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 50 of its 76 references that are in the map. First posted on arXiv 18 Sep 2026.

Builds on

50 of 76 references.

Closest ancestors · 15

Same direction, newest first

  1. 2025OjaKV: Context-Aware Online Low-Rank KV Cache Compression25 Sep
  2. 2025Jenga: Effective Memory Management for Serving LLM with Heterogeneity24 Mar
  3. 2025Towards Economical Inference: Enabling DeepSeek's Multi-Head Latent Attention in Any Transformer-based LLMs20 Feb
  4. 2025RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression19 Feb
  5. 2024A Survey on Large Language Model Acceleration based on KV Cache Management27 Dec
  6. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference28 Oct
  7. 2024LoRC: Low-Rank Compression for LLMs KV Cache with a Progressive Compression Strategy4 Oct
  8. 2024Compute Or Load KV Cache? Why Not Both?4 Oct
  9. 2024Eigen Attention: Attention in Low-Rank Space for KV Cache Compression10 Aug
  10. 2024Palu: Compressing KV-Cache with Low-Rank Projection30 Jul
  11. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management28 Jun
  12. 2024PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference21 May
3 more
  1. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  2. 2024Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inference14 Mar
  3. 2024Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inference14 Feb

Other references · 35

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2025Qwen2.5-VL Technical Reportcited by 5,418
  3. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  4. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  5. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  6. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  7. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  8. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  9. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
  10. 2024TinyLlama: An Open-Source Small Language Modelcited by 604
  11. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 507
  12. 2024Jamba: A Hybrid Transformer-Mamba Language Modelcited by 390
23 more
  1. 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 255
  2. 2024SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compressioncited by 231
  3. 2022Language model compression with weighted low-rank factorizationcited by 192
  4. 2023LLM in a flash: Efficient Large Language Model Inference with Limited Memorycited by 170
  5. 2024LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMscited by 142
  6. 2024Improve Vision Language Model Chain-of-thought Reasoningcited by 135
  7. 2024Zamba: A Compact 7B SSM Hybrid Modelcited by 118
  8. 2024LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelismcited by 110
  9. 2023LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximationcited by 103
  10. 2024PowerInfer-2: Fast Large Language Model Inference on a Smartphonecited by 101
  11. 2024Hydragen: High-Throughput LLM Inference with Shared Prefixescited by 71
  12. 2025SVD-LLM V2: Optimizing Singular Value Truncation for Large Language Model Compressioncited by 63
  13. 2024Fast On-device LLM Inference with NPUscited by 58
  14. 2023MR-GSM8K: A Meta-Reasoning Benchmark for Large Language Model Evaluationcited by 57
  15. 2024ALISA: Accelerating Large Language Model Inference via Sparsity-Aware KV Cachingcited by 48
  16. 2025Characterizing Mobile SoC for Accelerating Heterogeneous LLM Inferencecited by 20
  17. 2024Transformer-Lite: High-efficiency Deployment of Large Language Models on Mobile Phone GPUscited by 17
  18. 2025IC-Cache: Efficient Large Language Model Serving via In-context Cachingcited by 13
  19. 2024SmartMem: Layout Transformation Elimination and Adaptation for Efficient DNN Execution on Mobilecited by 12
  20. 2025MobileRAG: A Fast, Memory-Efficient, and Energy-Efficient Method for On-Device RAGcited by 9
  21. 2026Predicting LLM Output Length via Entropy-Guided Representationscited by 9
  22. 2025MNN-LLM: A Generic Inference Engine for Fast Large Language Model Deployment on Mobile Devicescited by 7
  23. 2026FlashMem: Supporting Modern DNN Workloads on Mobile with GPU Memory Hierarchy Optimizationscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026TTKV: Temporal-Tiered KV Cache for Long-Context LLM Inference
  2. 2026KVDrive: A Holistic Multi-Tier KV Cache Management System for Long-Context LLM Inference
  3. 2026Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimization
  4. 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inference
  5. 2026Don't Waste Bits! Adaptive KV-Cache Quantization for Lightweight On-Device LLMs
  6. 2025TailorKV: A Hybrid Framework for Long-Context Inference via Tailored KV Cache Optimization
  7. 2025PackKV: Reducing KV Cache Memory Footprint through LLM-Aware Lossy Compression
  8. 2025BaKlaVa -- Budgeted Allocation of KV cache for Long-context Inference