Selective KV Cache Protection for Noise-Resilient LLM Inference on Analog Compute-In-Memory Systems
On the map: Efficient LLM inference and serving → KV cache optimization
Abstract (arXiv)
Analog compute-in-memory (CIM) arrays have emerged as a promising substrate for energy-efficient LLM inference, particularly for weight-stationary computations in linear layers. However, extending analog CIM to attention mechanisms introduces a fundamental challenge: KV cache operations demand repeated in-situ weight updates, and the resulting mismatch with the weight-stationary paradigm exposes dynamic computations to significant hardware noise, a critical problem that remains largely unexplored. In this paper, we present the first systematic study of dynamic attention computation on analog CIM arrays, revealing that initial and recent tokens exhibit disproportionate vulnerability to hardware noise. Motivated by this token-level insight, we propose a hierarchical token protection strategy that keeps sink tokens and a sliding recent-token window on a higher-precision digital path while processing the bulk KV cache on analog CIM. A co-designed scheduler combines analog programming, ownership transition, and bulk-MVM tile formation to bound digital overhead. Evaluations on nine LLMs show that our approach lowers average perplexity under analog noise from 33.91 to 11.95, close to the clean baseline of 11.06, while improving dynamic-KV programming-row utilization from 23.1% to 91.2%.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 15 of its 37 references that are in the map. First posted on arXiv 31 Jul 2026.
Builds on
15 of 37 references.
Closest ancestors · 3
Same direction, newest first
- 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
- 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time26 May
Other references · 12
- 2017Attention Is All You Needcited by 60,361
- 2025Qwen3 Technical Reportcited by 8,820
- 2020Longformer: The Long-Document Transformercited by 3,413
- 2023Qwen Technical Reportcited by 3,344
- 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
- 2024OLMo: Accelerating the Science of Language Modelscited by 613
- 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
- 2023Hardware-aware training for large-scale and diverse deep learning inference workloads using in-memory computing-based acceleratorscited by 43
- 2022Computing-In-Memory Neural Network Accelerators for Safety-Critical Systems: Can Small Device Variations Be Disastrous?cited by 12
- 2025HPD: Hybrid Projection Decomposition for Robust State Space Models on Analog CIM Hardwarecited by 3
- 2025Extending Straight-Through Estimation for Robust Neural Networks on Analog CIM Hardwarecited by 2
- 2024An Analog and Digital Hybrid Attention Accelerator for Transformers with Charge-based In-memory Computingcited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026NOVA-CIM: Noise- and Correlation-Tolerant Stochastic Interfaces for Analog Compute-in-Memory
- 2026CONF-KV: Confidence-Aware KV Cache Eviction with Mixed-Precision Storage for Long-Horizon LLM
- 2025Accelerating LLM Inference via Dynamic KV Cache Placement in Heterogeneous Memory System
- 2025Analog Foundation Models
- 2024MiniKV: Pushing the Limits of LLM Inference via 2-Bit Layer-Discriminative KV Cache
- 2026KVDrive: A Holistic Multi-Tier KV Cache Management System for Long-Context LLM Inference
- 2026NANQ: Noise-Floor-Aware Mixed-Precision Non-Uniform Quantization for Analog Compute-in-Memory
- 2026Don't Waste Bits! Adaptive KV-Cache Quantization for Lightweight On-Device LLMs