mlsanityData to 7 Oct 2026

Correctness Forensics for Batch Speculative Decoding: Diagnosing the Ragged Tensor Problem

arXiv 2510.22876 ↗First posted 26 Oct 2025cs.CL cs.AIcited by 3

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

Inference optimizations are routinely evaluated by throughput alone, without verifying output correctness. We conduct a forensic analysis of batch speculative decoding and find that several widely-used implementations silently produce corrupted outputs (repetitive tokens, <unk> symbols) while reporting competitive speed; failures invisible to metrics like ROUGE. We trace the root cause to the ragged tensor problem: variable token acceptance desynchronizes position IDs, attention masks, and KV-cache across a batch. We formalize the synchronization invariants (rectangular alignment and position-ID contiguity) that valid batched inference must preserve and show that maintaining them incurs superlinear alignment overhead under contiguous layouts. EQSPEC enforces the invariants without custom kernels; EXSPEC schedules same-length sequences to bypass realignment. On SpecBench across three model families, EXSPEC reaches 3 x throughput at batch size 8 with 95% exact match to standard decoding; residual divergence traces to floating-point non-determinism, not synchronization error. Code:https://github.com/eBay/spec_dec

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 25 of its 39 references that are in the map. First posted on arXiv 26 Oct 2025.

Builds on

25 of 39 references.

Closest ancestors · 17

Same direction, newest first

  1. 2025EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test3 Mar
  2. 2025TETRIS: Optimal Draft Token Selection for Batch Speculative Decoding21 Feb
  3. 2024The N-Grammys: Accelerating Autoregressive Inference with Learning-Free Batched Speculation6 Nov
  4. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees24 Jun
  5. 2024TurboSpec: Closed-loop Speculation Control System for Optimizing LLM Serving Goodput20 Jun
  6. 2024BASS: Batched Attention-optimized Speculative Sampling24 Apr
  7. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decoding3 Feb
  8. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty26 Jan
  9. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads19 Jan
  10. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding15 Jan
  11. 2023Accelerating LLM Inference with Staged Speculative Decoding8 Aug
  12. 2023Accelerating Transformer Inference for Translation via Parallel Decoding17 May
5 more
  1. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification16 May
  2. 2023Speculative Decoding with Big Little Decoder15 Feb
  3. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
  4. 2022Fast Inference from Transformers via Speculative Decoding30 Nov
  5. 2022Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation30 Mar

Other references · 8

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  3. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  4. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,096
  5. 2016Multi30K: Multilingual English-German Image Descriptionscited by 250
  6. 2023Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipelinecited by 86
  7. 2023SPEED: Speculative Pipelined Execution for Efficient Decodingcited by 47
  8. 2023Predictive Pipelined Decoding: A Compute-Latency Trade-off for Exact LLM Decodingcited by 40

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026K-Forcing: Joint Next-K-Token Decoding via Push-Forward Language Modelingcited by 1
  2. 2026Delay-Adaptive Speculation Control for Low-Latency Edge-Cloud LLM Inferencecited by 0
  3. 2026Accelerating Visual On-Policy Distillation with Batched Speculative Jacobi Rolloutscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026ECHO: Elastic Speculative Decoding with Sparse Gating for High-Concurrency Scenarios
  2. 2025Speculative Verification: Exploiting Information Gain to Refine Speculative Decoding
  3. 2026DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation
  4. 2026Revisiting Lossy Verification in Speculative Decoding: Mechanisms, Trade-offs, and Failure Modes
  5. 2026MineDraft: A Framework for Batch Parallel Speculative Decoding
  6. 2026LLM-42: Enabling Determinism in LLM Inference with Verified Speculation
  7. 2024Block Verification Accelerates Speculative Decoding
  8. 2025Speculative Decoding: Performance or Illusion?