mlsanityData to 14 Oct 2026

EdgeLLM-Nano: Auditing Polar-Guided Key-Cache Compression and Consistency-Anchored Pruning on a 33.6 MB On-Device Language Model

arXiv 2610.14475 ↗First posted 10 Oct 2026cs.CLcited by 0

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Deploying language models (LMs) on the edge presents unique memory, latency, and power challenges absent in high-performance computing. KV-cache compression emerges as a promising approach to address these challenges. However, the effectiveness of current KV-cache compression methods has not been thoroughly examined on edge devices. Moreover, we show that existing evaluation methods do not always involve the KV-cache under test. We propose EdgeLLM-Nano, a measurement study of KV-cache compression using a nano-scale decoder-only LM (transformer) testbed that has 8.39M parameters, i.e., 33.6 MB, to enable end-to-end examination of each backend, seed, and power sample. Our results give a negative verdict for a polar-guided key-cache quantizer (round-to-nearest (RTN) control causes an order of magnitude less damage), as well as a consistency-anchored progressive weight pruner (one-shot magnitude-based pruning performs as well as, if not better). We also introduce Long-Context Decode Fidelity (LCDF) evaluation, a token-by-token examination of teacher-forced log-losses using the KV-cache under test. Unlike perplexity that gives the same 5.461 score to all methods, LCDF distinguishes different KV-cache compression backends thanks to its paired statistics across seeds and effect sizes. Our companion Raspberry Pi 5 measurement shows all compression backends we tested to be less energy-efficient per token compared to uncompressed FP16 inference. Except for KIVI that is close to lossless in terms of LCDF, our findings indicate KV-cache compression can be improved. LCDF offers a simple and effective way of shedding light on KV-cache compression methods, whose negative impacts have been overlooked by perplexity.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 25 of its 28 references that are in the map. First posted on arXiv 10 Oct 2026.

Builds on

25 of 28 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025KVzip: Query-Agnostic KV Cache Compression with Context Reconstruction29 May
  2. 2025Rethinking Key-Value Cache Compression Techniques for Large Language Model Serving31 Mar
  3. 2025Key, Value, Compress: A Systematic Exploration of KV Cache Compression Techniques14 Mar
  4. 2025Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Models31 Jan
  5. 2024A Survey on Large Language Model Acceleration based on KV Cache Management27 Dec
  6. 2024Palu: Compressing KV-Cache with Low-Rank Projection30 Jul
  7. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr

Other references · 18

  1. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  2. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  3. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  4. 2020GLU Variants Improve Transformercited by 1,804
  5. 2019Root Mean Square Layer Normalizationcited by 1,480
  6. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  7. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  8. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 800
  9. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  10. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 634
  11. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  12. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 449
6 more
  1. 2022Confident Adaptive Language Modelingcited by 310
  2. 2021Accelerating Sparse Deep Neural Networkscited by 223
  3. 2024QJL: 1-Bit Quantized JL Transform for KV Cache Quantization with Zero Overheadcited by 52
  4. 2025RotateKV: Accurate and Robust 2-Bit KV Cache Quantization for LLMs via Outlier-Aware Adaptive Rotationscited by 46
  5. 2025PolarQuant: Leveraging Polar Transformation for Efficient Key Cache Quantization and Decoding Accelerationcited by 10
  6. 2025LLMPi: Optimizing LLMs for High-Throughput on Raspberry Picited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Titanus: Enabling KV Cache Pruning and Quantization On-the-Fly for LLM Acceleration
  2. 2026KVDiagnosis: A Diagnostic Benchmark for KV-Cache Compression in Long-Context Language Models
  3. 2026KV-CoRE: Benchmarking Data-Dependent Low-Rank Compressibility of KV-Caches in LLMs
  4. 2025EVICPRESS: Joint KV-Cache Compression and Eviction for Efficient LLM Serving
  5. 2025Kelle: Co-design KV Caching and eDRAM for Efficient LLM Serving in Edge Computing
  6. 2026VFold: Symmetry-Aware Cross-Layer Value Cache Compression
  7. 2026TurboAngle: Near-Lossless KV Cache Compression via Uniform Angle Quantization
  8. 2025Compactor: Calibrated Query-Agnostic KV Cache Compression with Approximate Leverage Scores