HiKV: Hierarchical Importance-Aware KV Cache with Hardware Acceleration for LLM Decoding
On the map: Efficient LLM inference and serving → KV cache optimization
Abstract (arXiv)
With the rapid adoption of long-context large language models (LLMs), the continuously growing KV cache during decoding has become the critical memory bottleneck. To tackle this challenge, we propose HiKV, a novel algorithm-hardware co-design that exploits KV cache redundancy through hierarchical importance awareness. Algorithmically, HiKV compresses the KV cache at two granularities: Stage I evicts unimportant tokens within a fixed budget, and Stage II further loads only the significant elements of each retained token, reaching compression ratios unattainable at a single granularity. Architecturally, we develop a dedicated accelerator centered on a reconfigurable importance sorter that switches between the distinct sorting datapaths each stage requires, unifying the two-stage acceleration in one circuit with minimal overhead. Evaluated on representative LLMs, HiKV achieves up to 7.95x speedup and 90% energy reduction in the attention computation over the vanilla KV cache baseline within negligible 1% accuracy loss. Under iso-accuracy constraints, HiKV outperforms state-of-the-art importance-based methods by achieving an additional 1.82~4.87x reduction in external memory accesses. These benefits are enabled by specialized hardware components that add only 8% to the system area.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 38 of its 58 references that are in the map. First posted on arXiv 24 Jul 2026.
Builds on
38 of 58 references.
Closest ancestors · 14
Same direction, newest first
- 2025VEDA: Efficient LLM Generation Through Voting-based KV Cache Eviction and Dataflow-flexible Accelerator1 Jul
- 2025UniCAIM: A Unified CAM/CIM Architecture with Static-Dynamic KV Cache Pruning for Efficient Long-Context LLM Inference10 Apr
- 2024A Survey on Large Language Model Acceleration based on KV Cache Management27 Dec
- 2024DynamicKV: Task-Aware Adaptive KV Cache Compression for Long Context LLMs19 Dec
- 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads14 Oct
- 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference16 Jul
- 2024KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approaches1 Jul
- 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management28 Jun
- 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
- 2024Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inference14 Mar
- 2023SparQ Attention: Bandwidth-Efficient LLM Inference8 Dec
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
2 more
Other references · 24
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2020Longformer: The Long-Document Transformercited by 3,413
- 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
- 2024Qwen2 Technical Reportcited by 2,120
- 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
- 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
- 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
- 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
- 2024A Survey on Efficient Inference for Large Language Modelscited by 205
- 2023MathCoder: Seamless Code Integration in LLMs for Enhanced Mathematical Reasoningcited by 156
- 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 152
- 2023LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language Modelscited by 124
12 more
- 2024FlightLLM: Efficient Large Language Model Inference with a Complete Mapping Flow on FPGAscited by 64
- 2024ALISA: Accelerating Large Language Model Inference via Sparsity-Aware KV Cachingcited by 48
- 2024SepLLM: Accelerate Large Language Models by Compressing One Segment into One Separatorcited by 41
- 2021DeepScaleTool : A Tool for the Accurate Estimation of Technology Scaling in the Deep-Submicron Eracited by 30
- 2024When to Stop? Towards Efficient Code Generation in LLMs with Excess Token Preventioncited by 21
- 2024EdgeLLM: A Highly Efficient CPU-FPGA Heterogeneous Edge Accelerator for Large Language Modelscited by 21
- 2024Anda: Unlocking Efficient LLM Inference with a Variable-Length Grouped Activation Data Formatcited by 9
- 2025SkipKV: Selective Skipping of KV Generation and Storage for Efficient Inference with Large Reasoning Modelscited by 5
- 2025CoDec: Prefix-Shared Decoding Kernel for LLMscited by 4
- 2024Token-Picker: Accelerating Attention in Text Generation with Minimized Memory Transfer via Probability Estimationcited by 3
- 2025Titanus: Enabling KV Cache Pruning and Quantization On-the-Fly for LLM Accelerationcited by 2
- 2025APT-LLM: Exploiting Arbitrary-Precision Tensor Core Computing for LLM Accelerationcited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026OasisKV: Scaling In-Decode KV Cache Beyond HBM with Lookahead Sparse Prefetching
- 2025FastKV: Decoupling of Context Reduction and KV Cache Compression for Prefill-Decoding Acceleration
- 2026BELT : BEhavior-Aligned Low-Rank KV Cache Compression with Learned Top-k Rank Allocation
- 2026HieraSparse: Hierarchical Semi-Structured Sparse KV Attention
- 2026KV-CoRE: Benchmarking Data-Dependent Low-Rank Compressibility of KV-Caches in LLMs
- 2026Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimization
- 2025HCAttention: Extreme KV Cache Compression via Heterogeneous Attention Computing for LLMs
- 2025PackKV: Reducing KV Cache Memory Footprint through LLM-Aware Lossy Compression