# CSV-Decode: Certifiable Sub-Vocabulary Decoding for Efficient Large Language Model Inference

arXiv: https://arxiv.org/abs/2511.21702 · HTML: https://mlsanity.com/paper/2511.21702/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving.
- Builds on 17 of its 25 references that are in the map.
- First posted on arXiv 16 Nov 2025.

## Placed in

- Efficient LLM inference and serving / [Speculative decoding](https://mlsanity.com/inference/speculative-decoding/)

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [Fast Inference from Transformers via Speculative Decoding](https://mlsanity.com/paper/2211.17192/) (2022)
- [Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads](https://mlsanity.com/paper/2401.10774/) (2024)
- [Reducing Transformer Depth on Demand with Structured Dropout](https://mlsanity.com/paper/1909.11556/) (2019)
- [Speculative Decoding with Big Little Decoder](https://mlsanity.com/paper/2302.07863/) (2023)
- [REST: Retrieval-Based Speculative Decoding](https://mlsanity.com/paper/2311.08252/) (2023)
- [PEARL: Parallel Speculative Decoding with Adaptive Draft Length](https://mlsanity.com/paper/2408.11850/) (2024)
- [Efficient softmax approximation for GPUs](https://mlsanity.com/paper/1609.04309/) (2016)
- [LLMEasyQuant: Scalable Quantization for Parallel and Distributed LLM Inference](https://mlsanity.com/paper/2406.19657/) (2024)
- [FastKV: Decoupling of Context Reduction and KV Cache Compression for Prefill-Decoding Acceleration](https://mlsanity.com/paper/2502.01068/) (2025)
- [TinyServe: Query-Aware Cache Selection for Efficient LLM Serving](https://mlsanity.com/paper/2509.12211/) (2025)
- [Designing Large Foundation Models for Efficient Training and Inference: A Survey](https://mlsanity.com/paper/2409.01990/) (2024)
- [PiKV: KV Cache Management System for Mixture of Experts](https://mlsanity.com/paper/2508.06526/) (2025)

## Built on by (most cited first, 7 of 7)

- [CXL-SpecKV: A Disaggregated FPGA Speculative KV-Cache for Datacenter LLM Serving](https://mlsanity.com/paper/2512.11920/) (2025)
- [Retrieval--Reasoning Processes for Multi-hop Question Answering: A Four-Axis Design Framework and Empirical Trends](https://mlsanity.com/paper/2601.00536/) (2026)
- [ConceptMoE: Adaptive Token-to-Concept Compression for Implicit Compute Allocation](https://mlsanity.com/paper/2601.21420/) (2026)
- [AdaCorrection: Adaptive Offset Cache Correction for Accurate Diffusion Transformers](https://mlsanity.com/paper/2602.13357/) (2026)
- [SoftWater: Class-Aware Rate Allocation for Softmax Quantization](https://mlsanity.com/paper/2608.12026/) (2026)
- [Thoughts-as-Planning: Latent World Models for Chain-of-Thoughts Optimization via Reinforcement Planning](https://mlsanity.com/paper/2605.28842/) (2026)
- [Two-Level Softmax Sampling Done Right: Correcting Bias from Size Imbalance and Dispersion](https://mlsanity.com/paper/2610.10483/) (2026)

## Abstract (arXiv)

Large language models face significant computational bottlenecks during inference due to the expensive output layer computation over large vocabularies. We present CSV-Decode, a novel approach that uses geometric upper bounds to construct small sub-vocabularies for each decoding step, enabling efficient sparse computation while maintaining dual correctness guarantees: exact top-k certification and varepsilon-certified softmax approximations. Our method clusters vocabulary embeddings offline and uses centroid-plus-radius bounds to identify which tokens can be safely omitted from computation. We provide a complete system implementation with sparse GEMV kernels, multi-GPU sharding, and CUDA Graph optimization. Experimental results demonstrate significant speedup over full vocabulary decoding while maintaining distributional guarantees and low fallback rates. Our code implementation available at hrefhttps://github.com/FastLM/CSV-Decodehttps://github.com/FastLM/CSV-Decode.