mlsanityData to 7 Oct 2026

SIMPLE: Disaggregating Sampling from GPU Inference into a Decision Plane for Faster Distributed LLM Serving

arXiv 2512.00719 ↗First posted 30 Nov 2025cs.DCcited by 1

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

As large language models (LLMs) scale out with tensor parallelism (TP) and pipeline parallelism (PP) and production stacks have aggressively optimized the data plane (attention/GEMM and KV cache), sampling, the decision plane that turns logits into tokens, becomes a new bottleneck. This creates a structural holdout: sampling neither expands with TP nor balances across PP stages, so its share of iteration time grows as GPUs get faster and it caps pipeline frequency at the last stage. We present SIMPLE, a stage-agnostic, sequence-parallel, overlappable decision plane that disaggregates sampling into a CPU-side service and shrinks its runtime footprint back to a minor, hidden role. SIMPLE combines: (1) sequence-parallel sampling, which shards work along the batch dimension and removes vocabulary-axis collectives; (2) a CPU-based algorithm with column-wise penalties and truncation-first filtering to realize single-pass, linear-time kernels; and (3) speculative hot-vocab sampling (SHVS), which samples on a small hot set with rejection-correctness and uses a simple sizing model to choose the hot-vocab size that maximizes throughput. In evaluation, SIMPLE improves end-to-end throughput by up to 96% and reduces P95 latency by 20-65%. Crucially, SIMPLE requires no user-side code changes and composes with existing data-plane optimizations, unlocking scaling benefits that compound with future GPU generations.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 24 of its 44 references that are in the map. First posted on arXiv 30 Nov 2025.

Builds on

24 of 44 references.

References · 24

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  5. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  6. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  7. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  8. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  9. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  10. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  11. 2018Hierarchical Neural Story Generationcited by 1,182
  12. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
12 more
  1. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 848
  2. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  3. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  4. 2025Kimi K2: Open Agentic Intelligencecited by 523
  5. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funnelingcited by 420
  6. 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Servingcited by 200
  7. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Servingcited by 199
  8. 2024Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloadscited by 120
  9. 2025FR-Spec: Accelerating Large-Vocabulary Language Models via Frequency-Ranked Speculative Samplingcited by 20
  10. 2025VOCABTRIM: Vocabulary Pruning for Efficient Speculative Decoding in LLMscited by 11
  11. 2025any4: Learned 4-bit Numeric Representation for LLMscited by 5
  12. 2025DynaSpec: Context-aware Dynamic Speculative Sampling for Large-Vocabulary Language Modelscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026FlashSampling: Fast and Memory-Efficient Exact Samplingcited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2026Characterizing Parallelism Strategies in LLM Inference: Fundamental Compute-Communication Trade-offs
  2. 2026Scaling LLM Inference Beyond Amdahl`s Limits via Eliminating Non-Scalable Overheads ↗
  3. 2025Striking the Right Balance between Compute and Copy: Improving LLM Inferencing Under Speculative Decoding
  4. 2026KernelSight-LM: A Kernel-Level LLM Inference Simulator
  5. 2025Shift Parallelism: Low-Latency, High-Throughput LLM Inference for Dynamic Workloads
  6. 2025Characterizing Communication Patterns in Distributed Large Language Model Inference
  7. 2025Characterizing LLM Inference Energy-Performance Tradeoffs across Workloads and GPU Scaling
  8. 2025Efficient LLM Inference with Activation Checkpointing and Hybrid Caching