# Leveraging Speculative Sampling and KV-Cache Optimizations Together for Generative AI using OpenVINO

arXiv: https://arxiv.org/abs/2311.04951 · HTML: https://mlsanity.com/paper/2311.04951/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving.
- Builds on 5 of its 11 references that are in the map.
- First posted on arXiv 8 Nov 2023.

## Placed in

- Efficient LLM inference and serving / [Speculative decoding](https://mlsanity.com/inference/speculative-decoding/)

## Builds on (most cited first, 5 of 5)

- [Accelerating Large Language Model Decoding with Speculative Sampling](https://mlsanity.com/paper/2302.01318/) (2023)
- [Eliciting Latent Predictions from Transformers with the Tuned Lens](https://mlsanity.com/paper/2303.08112/) (2023)
- [Efficiently Scaling Transformer Inference](https://mlsanity.com/paper/2211.05102/) (2022)
- [Blockwise Parallel Decoding for Deep Autoregressive Models](https://mlsanity.com/paper/1811.03115/) (2018)
- [Confident Adaptive Language Modeling](https://mlsanity.com/paper/2207.07061/) (2022)

## Built on by (most cited first, 1 of 1)

- [Accelerated AI Inference via Dynamic Execution Methods](https://mlsanity.com/paper/2411.00853/) (2024)

## Abstract (arXiv)

Inference optimizations are critical for improving user experience and reducing infrastructure costs and power consumption. In this article, we illustrate a form of dynamic execution known as speculative sampling to reduce the overall latency of text generation and compare it with standard autoregressive sampling. This can be used together with model-based optimizations (e.g. quantization) to provide an optimized solution. Both sampling methods make use of KV caching. A Jupyter notebook and some sample executions are provided.