mlsanityData to 14 Oct 2026

Leveraging Speculative Sampling and KV-Cache Optimizations Together for Generative AI using OpenVINO

arXiv 2311.04951 ↗First posted 8 Nov 2023cs.LG cs.AI cs.PFcited by 1

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

Inference optimizations are critical for improving user experience and reducing infrastructure costs and power consumption. In this article, we illustrate a form of dynamic execution known as speculative sampling to reduce the overall latency of text generation and compare it with standard autoregressive sampling. This can be used together with model-based optimizations (e.g. quantization) to provide an optimized solution. Both sampling methods make use of KV caching. A Jupyter notebook and some sample executions are provided.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 5 of its 11 references that are in the map. First posted on arXiv 8 Nov 2023.

Builds on

5 of 11 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb

Other references · 4

  1. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 623
  2. 2022Efficiently Scaling Transformer Inferencecited by 539
  3. 2018Blockwise Parallel Decoding for Deep Autoregressive Modelscited by 323
  4. 2022Confident Adaptive Language Modelingcited by 310

Built on by

Cited by year

Citing papers per year: 2024: 1.

1 paper.

  1. 2024Accelerated AI Inference via Dynamic Execution Methodscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023S3: Increasing GPU Utilization during Generative Inference for Higher Throughput
  2. 2025READER: Retrieval-Assisted Drafter for Efficient LLM Inference
  3. 2026TAPS: Task Aware Proposal Distributions for Speculative Sampling
  4. 2026Speculative Sampling with Reinforcement Learning
  5. 2025Accelerated Diffusion Models via Speculative Sampling
  6. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models
  7. 2024BASS: Batched Attention-optimized Speculative Sampling
  8. 2024The N-Grammys: Accelerating Autoregressive Inference with Learning-Free Batched Speculation