mlsanityData to 7 Oct 2026

Striking the Right Balance between Compute and Copy: Improving LLM Inferencing Under Speculative Decoding

arXiv 2511.12031 ↗First posted 15 Nov 2025cs.DC cs.AIcited by 0

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

With the skyrocketing costs of GPUs and their virtual instances in the cloud, there is a significant desire to use CPUs for large language model (LLM) inference. KV cache update, often implemented as allocation, copying, and in-place strided update for each generated token, incurs significant overhead. As the sequence length increases, the allocation and copy overheads dominate the performance. Alternate approaches may allocate large KV tensors upfront to enable in-place updates, but these matrices (with zero-padded rows) cause redundant computations. In this work, we propose a new KV cache allocation mechanism called Balancing Memory and Compute (BMC). BMC allocates, once every r iterations, KV tensors with r redundant rows, allowing in-place update without copy overhead for those iterations, but at the expense of a small amount of redundant computation. Second, we make an interesting observation that the extra rows allocated in the KV tensors and the resulting redundant computation can be repurposed for Speculative Decoding (SD) that improves token generation efficiency. Last, BMC represents a spectrum of design points with different values of r. To identify the best-performing design point(s), we derive a simple analytical model for BMC. The proposed BMC method achieves an average throughput acceleration of up to 3.2x over baseline HuggingFace (without SD). Importantly when we apply BMC with SD, it results in an additional speedup of up to 1.39x, over and above the speedup offered by SD. Further, BMC achieves a throughput acceleration of up to 1.36x and 2.29x over state-of-the-art inference servers vLLM and DeepSpeed, respectively. Although the BMC technique is evaluated extensively across different classes of CPUs (desktop and server class), we also evaluate the scheme with GPUs and demonstrate that it works well for GPUs.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 27 of its 59 references that are in the map. First posted on arXiv 15 Nov 2025.

Builds on

27 of 59 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs3 Oct
  2. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun

Other references · 25

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  4. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  5. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  6. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  7. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  8. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  9. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  10. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  11. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  12. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 564
13 more
  1. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 507
  2. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  3. 2023A Survey on Model Compression for Large Language Modelscited by 263
  4. 2024Sequoia: Scalable, Robust, and Hardware-aware Speculative Decodingcited by 103
  5. 2024vAttention: Dynamic Memory Management for Serving LLMs without PagedAttentioncited by 93
  6. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 62
  7. 2024A Survey on Transformer Compressioncited by 44
  8. 2025PARD: Accelerating LLM Inference with Low-Cost PARallel Draft Model Adaptationcited by 28
  9. 2023Efficient LLM Inference on CPUscited by 24
  10. 2021Accelerating SLIDE Deep Learning on Modern CPUs: Vectorization, Quantizations, Memory Optimizations, and Morecited by 15
  11. 2024A Comprehensive Survey of Compression Algorithms for Language Modelscited by 15
  12. 2020Automatic Kernel Generation for Volta Tensor Corescited by 8
  13. 2024Inference Performance Optimization for Large Language Models on CPUscited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference
  2. 2024KVPR: Efficient LLM Inference with I/O-Aware KV Cache Partial Recomputation
  3. 2025Efficient LLM Inference with Activation Checkpointing and Hybrid Caching
  4. 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inference
  5. 2026Comparative Characterization of KV Cache Management Strategies for LLM Inference
  6. 2024XKV: Personalized KV Cache Memory Reduction for Long-Context LLM Inference
  7. 2025SpeCache: Speculative Key-Value Caching for Efficient Generation of LLMs
  8. 2026VeriCache: Turning Lossy KV Cache into Lossless LLM Inference