mlsanityData to 7 Oct 2026

CSV-Decode: Certifiable Sub-Vocabulary Decoding for Efficient Large Language Model Inference

arXiv 2511.21702 ↗First posted 16 Nov 2025cs.CL cs.AIcited by 7

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

Large language models face significant computational bottlenecks during inference due to the expensive output layer computation over large vocabularies. We present CSV-Decode, a novel approach that uses geometric upper bounds to construct small sub-vocabularies for each decoding step, enabling efficient sparse computation while maintaining dual correctness guarantees: exact top-k certification and varepsilon-certified softmax approximations. Our method clusters vocabulary embeddings offline and uses centroid-plus-radius bounds to identify which tokens can be safely omitted from computation. We provide a complete system implementation with sparse GEMV kernels, multi-GPU sharding, and CUDA Graph optimization. Experimental results demonstrate significant speedup over full vocabulary decoding while maintaining distributional guarantees and low fallback rates. Our code implementation available at hrefhttps://github.com/FastLM/CSV-Decodehttps://github.com/FastLM/CSV-Decode.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 17 of its 25 references that are in the map. First posted on arXiv 16 Nov 2025.

Builds on

17 of 25 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024PEARL: Parallel Speculative Decoding with Adaptive Draft Length13 Aug
  2. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads19 Jan
  3. 2023REST: Retrieval-Based Speculative Decoding14 Nov
  4. 2023Speculative Decoding with Big Little Decoder15 Feb
  5. 2022Fast Inference from Transformers via Speculative Decoding30 Nov

Other references · 12

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  4. 2023Mistral 7Bcited by 3,065
  5. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  6. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 434
  7. 2016Efficient softmax approximation for GPUscited by 67
  8. 2024LLMEasyQuant: Scalable Quantization for Parallel and Distributed LLM Inferencecited by 15
  9. 2025FastKV: Decoupling of Context Reduction and KV Cache Compression for Prefill-Decoding Accelerationcited by 13
  10. 2025TinyServe: Query-Aware Cache Selection for Efficient LLM Servingcited by 9
  11. 2024Designing Large Foundation Models for Efficient Training and Inference: A Surveycited by 7
  12. 2025PiKV: KV Cache Management System for Mixture of Expertscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 6.

7 papers.

  1. 2025CXL-SpecKV: A Disaggregated FPGA Speculative KV-Cache for Datacenter LLM Servingcited by 8
  2. 2026Retrieval--Reasoning Processes for Multi-hop Question Answering: A Four-Axis Design Framework and Empirical Trendscited by 8
  3. 2026AdaCorrection: Adaptive Offset Cache Correction for Accurate Diffusion Transformerscited by 3
  4. 2026ConceptMoE: Adaptive Token-to-Concept Compression for Implicit Compute Allocationcited by 2
  5. 2026SoftWater: Class-Aware Rate Allocation for Softmax Quantizationcited by 2
  6. 2026Thoughts-as-Planning: Latent World Models for Chain-of-Thoughts Optimization via Reinforcement Planningcited by 1
  7. 2026Two-Level Softmax Sampling Done Right: Correcting Bias from Size Imbalance and Dispersioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Chimera: A Lossless Decoding Method for Accelerating Large Language Models Inference by Fusing all Tokens
  2. 2026SparseDecoding: Decoding-Aware Pruning for Accurate and Efficient LLM Inference
  3. 2024Q-Sparse: All Large Language Models can be Fully Sparsely-Activated
  4. 2024Generation Meets Verification: Accelerating Large Language Model Inference with Smart Parallel Auto-Correct Decoding
  5. 2025Free Draft-and-Verification: Toward Lossless Parallel Decoding for Diffusion Large Language Models
  6. 2024CharED: Character-wise Ensemble Decoding for Large Language Models
  7. 2026Decoding as Optimisation on the Probability Simplex: From Top-K to Top-P (Nucleus) to Best-of-K Samplers
  8. 2026Beyond Tokens: A Survey on Decoding Methods for Large Language and Vision-Language Models