mlsanityData to 14 Oct 2026

Kelle: Co-design KV Caching and eDRAM for Efficient LLM Serving in Edge Computing

arXiv 2510.16040 ↗First posted 16 Oct 2025cs.AR cs.AIcited by 5

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Running Large Language Models (LLMs) on edge devices is crucial for reducing latency, improving real-time processing, and enhancing privacy. By performing inference directly on the device, data does not need to be sent to the cloud, ensuring faster responses and reducing reliance on network connectivity. However, implementing LLMs on edge devices presents challenges, particularly with managing key-value (KV) caches, which plays a pivotal role in LLM serving. As the input text lengthens, the size of the KV cache increases linearly with the sequence length, leading to a significant memory footprint and data access costs. On the other hand, edge devices have limited memory and computational power, making it hard to store and efficiently access the large caches needed for LLM inference. To mitigate the substantial overhead caused by KV cache, we propose using embedded DRAM (eDRAM) as the primary storage for LLM serving in edge device, which offers higher storage density compared to SRAM. However, to ensure data integrity, eDRAM needs periodic refresh operations, which are power-intensive. To reduce eDRAM costs and improve overall system performance, we propose~textitKelle, a software-hardware co-design solution optimized for deploying LLMs on eDRAM-based edge systems. Combined with our fine-grained memory eviction, recomputation, and refresh control algorithms, the textitKelle accelerator delivers a 3.9× speedup and 4.5× energy savings compared to existing baseline solutions.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 40 of its 89 references that are in the map. First posted on arXiv 16 Oct 2025.

Builds on

40 of 89 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management28 Jun
  2. 2024PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference21 May
  3. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time26 May

Other references · 37

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2025Qwen2.5-VL Technical Reportcited by 5,587
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  5. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  6. 2016Pointer Sentinel Mixture Modelscited by 3,096
  7. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  8. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  9. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  10. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  11. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  12. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
25 more
  1. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 812
  2. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 726
  3. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 701
  4. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  5. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 634
  6. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  7. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  8. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 486
  9. 2021Pay Attention to MLPscited by 368
  10. 2020SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruningcited by 234
  11. 2020GOBO: Quantizing Attention-Based NLP Models for Low Latency and Energy Efficient Inferencecited by 106
  12. 2023OliVe: Accelerating Large Language Models via Hardware-friendly Outlier-Victim Pair Quantizationcited by 92
  13. 2024Fast On-device LLM Inference with NPUscited by 62
  14. 2021Softermax: Hardware/Software Co-Design of an Efficient Softmax for Transformerscited by 54
  15. 2020Hardware Accelerator for Multi-Head Attention and Position-Wise Feed-Forward in the Transformercited by 43
  16. 2021FAST: DNN Training Under Variable Precision Block Floating Point with Stochastic Roundingcited by 31
  17. 2024Tender: Accelerating Large Language Models via Tensor Decomposition and Runtime Requantizationcited by 28
  18. 2024EDGE-LLM: Enabling Efficient Large Language Model Adaptation on Edge Devices via Layerwise Unified Compression and Adaptive Layer Tuning and Votingcited by 27
  19. 2025Speculative Decoding and Beyond: An In-Depth Survey of Techniquescited by 23
  20. 2024Cambricon-LLM: A Chiplet-Based Hybrid Architecture for On-Device Inference of 70B LLMcited by 18
  21. 2024DFRot: Achieving Outlier-Free and Massive Activation-Free for Rotated LLMs with Refined Rotationcited by 17
  22. 2025DREAM: Drafting with Refined Target Features and Entropy-Adaptive Cross-Attention Fusion for Multimodal Speculative Decodingcited by 14
  23. 2024COMET: Towards Partical W4A4KV4 LLMs Servingcited by 6
  24. 2023Hyft: A Reconfigurable Softmax Accelerator with Hybrid Numeric Format for both Training and Inferencecited by 4
  25. 2025PipeSpec: Breaking Stage Dependencies in Hierarchical LLM Decodingcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026Agent Memory Below the Prompt: Persistent Q4 KV Cache for Multi-Agent LLM Inference on Edge Devicescited by 8
  2. 2026SHIELD: A Segmented Hierarchical Memory Architecture for Energy-Efficient LLM Inference on Edge NPUscited by 0
  3. 2026Bit-Flip Vulnerability of Shared KV-Cache Blocks in LLM Serving Systemscited by 0
  4. 2026Beyond Prefill-Decode Disaggregation: Dissecting LLM Inference for Heterogeneous Platforms via Dynamic Operator Schedulingcited by 0
  5. 2026Unified AI Gateway: A Framework for Joint Model Routing and KV Cache Managementcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026PALUTE: Processing-In-Memory Acceleration via Lookup Table for Edge LLM Inference
  2. 2026EdgeLLM-Nano: Auditing Polar-Guided Key-Cache Compression and Consistency-Anchored Pruning on a 33.6 MB On-Device Language Model
  3. 2025VEDA: Efficient LLM Generation Through Voting-based KV Cache Eviction and Dataflow-flexible Accelerator
  4. 2025AdaptCache: KV Cache Native Storage Hierarchy for Low-Delay and High-Quality Language Model Serving
  5. 2025EdgeLoRA: An Efficient Multi-Tenant LLM Serving System on Edge Devices
  6. 2025Accelerating LLM Inference via Dynamic KV Cache Placement in Heterogeneous Memory System
  7. 2026KVDrive: A Holistic Multi-Tier KV Cache Management System for Long-Context LLM Inference
  8. 2026Hardware-Software Co-design for 3D-DRAM-based LLM Serving Accelerator