mlsanityData to 14 Oct 2026

SpecFit: Not All Draft Tokens Cost the Same in Offloaded MoE Inference

arXiv 2610.15417 ↗First posted 11 Oct 2026cs.AIcited by 0

On the map: Mixture-of-experts → Efficient MoE training and inference systems

also in 1 more
Abstract (arXiv)

Mixture-of-experts (MoE) models activate only a few experts per token, but when their expert weights are offloaded to host memory, host-to-device (H2D) expert transfers enter the critical path of decoding. Speculative decoding verifies several draft tokens in one forward pass, yet in an MoE model each verified token activates its own experts, so a larger draft tree transfers more experts. Because of this H2D cost, speculative decoding commits several tokens per verification cycle yet is sometimes even slower than autoregressive decoding. Based on this observation, we propose SpecFit, which chooses the verified draft subtree to maximize predicted throughput rather than acceptance length. SpecFit estimates the expected number of committed tokens from draft path probabilities and predicts cycle time from the node count and the per-layer union of the experts the subtree activates; this union is a monotone submodular coverage function, so each node pays only for the experts it adds. An exact search over ancestor-closed subtrees returns the subtree with the highest predicted throughput; run after each draft level, the same search also decides when to stop drafting, and the unmodified target verifies the result. On CNN/DailyMail summarization and a GSM8K subset with Qwen3-30B-A3B, SpecFit decodes 21.6% and 7.0% faster than EAGLE-3 with the same expert cache and 1.97x and 1.72x as fast as vanilla EAGLE-3. Code is available at https://github.com/Chlience/SpecFit.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 21 of its 29 references that are in the map. First posted on arXiv 11 Oct 2026.

Builds on

21 of 29 references.

Closest ancestors · 7

Same direction, newest first

  1. 2026DraftExpert: Expansion-Aware Self-Speculative Decoding for End-Device MoE Inference27 Jul
  2. 2026MoE-Spec: Expert Budgeting for Efficient Speculative Decoding17 Feb
  3. 2025SP-MoE: Speculative Decoding and Prefetching for Accelerating MoE-based Model Inference11 Oct
  4. 2024ProMoE: Fast MoE-based LLM Serving using Proactive Caching29 Oct
  5. 2024Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models10 Feb
  6. 2024MoE-Infinity: Efficient MoE Inference on Personal Machines with Sparsity-Aware Expert Cache25 Jan
  7. 2023Fast Inference of Mixture-of-Experts Language Models with Offloading28 Dec

Other references · 14

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  4. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  5. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 1,006
  6. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 812
  7. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Treescited by 370
  8. 2025EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Testcited by 331
  9. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decodingcited by 269
  10. 2024OPT-Tree: Speculative Decoding with Adaptive Draft Tree Structurecited by 67
  11. 2024SpecExec: Massively Parallel Speculative Decoding for Interactive LLM Inference on Consumer Devicescited by 46
  12. 2026ETS: Energy-Guided Test-Time Scaling for Training-Free RL Alignmentcited by 5
2 more
  1. 2025Pre-Attention Expert Prediction and Prefetching for Mixture-of-Experts Large Language Modelscited by 4
  2. 2026Turning Off-Policy Tokens On-Policy: A Plug-in Approach for Improving LLM Alignmentcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Less Experts, Faster Decoding: Cost-Aware Speculative Decoding for Mixture-of-Experts
  2. 2026AcceptMoE: Commitment-Weighted Self-Sizing Verifier Expert Sets for Efficient MoE Speculative Decoding
  3. 2026Efficient Mixture-of-Experts with Speculative Decoding via Expert Coactivation
  4. 2026Making Every Verified Token Count: Adaptive Verification for MoE Speculative Decoding
  5. 2025Accelerating Mixture-of-Experts Inference by Hiding Offloading Latency with Speculative Decoding
  6. 2025MoE-SpeQ: Speculative Quantized Decoding with Proactive Expert Prefetching and Offloading for Mixture-of-Experts
  7. 2026The Limits of Speculation: Bounding Speculative Decoding in Mixture-of-Experts
  8. 2026Speculating Experts Accelerates Inference for Mixture-of-Experts