VEDA: Efficient LLM Generation Through Voting-based KV Cache Eviction and Dataflow-flexible Accelerator
On the map: Efficient LLM inference and serving → KV cache optimization
Abstract (arXiv)
Large Language Models (LLMs) excel in natural language processing tasks but pose significant computational and memory challenges for edge deployment due to their intensive resource demands. This work addresses the efficiency of LLM inference by algorithm-hardware-dataflow tri-optimizations. We propose a novel voting-based KV cache eviction algorithm, balancing hardware efficiency and algorithm accuracy by adaptively identifying unimportant kv vectors. From a dataflow perspective, we introduce a flexible-product dataflow and a runtime reconfigurable PE array for matrix-vector multiplication. The proposed approach effectively handles the diverse dimensional requirements and solves the challenges of incrementally varying sequence lengths. Additionally, an element-serial scheduling scheme is proposed for nonlinear operations, such as softmax and layer normalization (layernorm). Results demonstrate a substantial reduction in latency, accompanied by a significant decrease in hardware complexity, from O(N) to O(1). The proposed solution is realized in a custom-designed accelerator, VEDA, which outperforms existing hardware platforms. This research represents a significant advancement in LLM inference on resource-constrained edge devices, facilitating real-time processing, enhancing data privacy, and enabling model customization.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 16 of its 21 references that are in the map. First posted on arXiv 1 Jul 2025.
Builds on
16 of 21 references.
Closest ancestors · 3
Same direction, newest first
- 2024Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inference14 Mar
- 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun
- 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time26 May
Other references · 13
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
- 2023GPT-4 Technical Reportcited by 18,396
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
- 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
- 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
- 2019Compressive Transformers for Long-Range Sequence Modellingcited by 726
- 2020SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruningcited by 234
- 2018Online normalizer calculation for softmaxcited by 162
- 2020A3: Accelerating Attention Mechanisms in Neural Networks with Approximationcited by 101
- 2021Softermax: Hardware/Software Co-Design of an Efficient Softmax for Transformerscited by 54
- 2021DeepScaleTool : A Tool for the Accurate Estimation of Technology Scaling in the Deep-Submicron Eracited by 30
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inference
- 2025Accelerating LLM Inference via Dynamic KV Cache Placement in Heterogeneous Memory System
- 2023Efficient LLM inference solution on Intel GPU
- 2025Titanus: Enabling KV Cache Pruning and Quantization On-the-Fly for LLM Acceleration
- 2024A Survey on Large Language Model Acceleration based on KV Cache Management
- 2024KVPR: Efficient LLM Inference with I/O-Aware KV Cache Partial Recomputation
- 2024XKV: Personalized KV Cache Memory Reduction for Long-Context LLM Inference
- 2024Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumption