mlsanityData to 7 Oct 2026

SpecForge: A Flexible and Efficient Open-Source Training Framework for Speculative Decoding

arXiv 2603.18567 ↗First posted 19 Mar 2026cs.LG cs.AI cs.CLcited by 15

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

Large language models incur high inference latency due to sequential autoregressive decoding. Speculative decoding alleviates this bottleneck by using a lightweight draft model to propose multiple tokens for batched verification. However, its adoption has been limited by the lack of high-quality draft models and scalable training infrastructure. We introduce SpecForge, an open-source, production-oriented framework for training speculative decoding models with full support for EAGLE-3. SpecForge incorporates target-draft decoupling, hybrid parallelism, optimized training kernels, and integration with production-grade inference engines, enabling up to 9.9x faster EAGLE-3 training for Qwen3-235B-A22B. In addition, we release SpecBundle, a suite of production-grade EAGLE-3 draft models trained with SpecForge for mainstream open-source LLMs. Through a systematic study of speculative decoding training recipes, SpecBundle addresses the scarcity of high-quality drafts in the community, and our draft models achieve up to 4.48x end-to-end inference speedup on SGLang, establishing SpecForge as a practical foundation for real-world speculative decoding deployment.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 29 of its 38 references that are in the map. First posted on arXiv 19 Mar 2026.

Builds on

29 of 38 references.

Closest ancestors · 15

Same direction, newest first

  1. 2025EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test3 Mar
  2. 2025Speculative Decoding and Beyond: An In-Depth Survey of Techniques27 Feb
  3. 2024SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Acceleration9 Oct
  4. 2024Learning Harmonized Representations for Speculative Sampling28 Aug
  5. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees24 Jun
  6. 2024GliDe with a CaPE: A Low-Hassle Method to Accelerate Speculative Decoding3 Feb
  7. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decoding3 Feb
  8. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty26 Jan
  9. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads19 Jan
  10. 2023SpecTr: Fast Speculative Decoding via Optimal Transport23 Oct
  11. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification16 May
  12. 2023Speculative Decoding with Big Little Decoder15 Feb
3 more
  1. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
  2. 2022Fast Inference from Transformers via Speculative Decoding30 Nov
  3. 2022Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation30 Mar

Other references · 14

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  2. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  3. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  4. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  5. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  6. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  7. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 641
  8. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  9. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 507
  10. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Servingcited by 248
  11. 2025MTBench: A Multimodal Time Series Benchmark for Temporal Reasoning and Question Answeringcited by 50
  12. 2024The Perfect Blend: Redefining RLHF with Mixture of Judgescited by 43
2 more
  1. 2025Mind the Memory Gap: Unveiling GPU Bottlenecks in Large-Batch LLM Inferencecited by 32
  2. 2025FinanceQA: A Benchmark for Evaluating Financial Analysis Capabilities of Large Language Modelscited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 15.

15 papers.

Sort:

By area:

  1. 2026D-PACE: Dynamic Position-Aware Cross-Entropy for Parallel Speculative Draftingcited by 16
  2. 2026Draft-OPD: On-Policy Distillation for Speculative Draft Modelscited by 10
  3. 2026Making Every Verified Token Count: Adaptive Verification for MoE Speculative Decodingcited by 3
  4. 2026Online Draft Co-Training for Speculative Decoding in Large-Scale, Long-Context RL Post-Trainingcited by 2
  5. 2026LoopSpec: Pipelined Self-Speculative Decoding for Looped Transformerscited by 2
  6. 2026DBLAST: Dependent Block Drafting for Stochastic Speculative Decodingcited by 1
  7. 2026PSD: Pushing the Pareto Frontier of Diffusion LLMs via Parallel Speculative Decodingcited by 0
  8. 2026LibraSpec: Dynamic Diffusion-Based Speculative Decoding via Marginal-Gain-Driven Optimizationcited by 0
  9. 2026Is Multimodal Speculative Decoding Ready for Diffusion-Based Parallel Drafting? A Survey and Empirical Diagnosiscited by 0
  10. 2026Vision Is Not Overhead: One-Pass Block Drafting for Lossless Speculative Decoding in Vision-Language Modelscited by 0
  11. 2026OUTLETS: Output-Length Prediction from Speculative Decoding Backbonescited by 0
  12. 2026Block Parallelism For Efficient Distributed Long-Context Diffusion Language Model Trainingcited by 0
3 more
  1. 2026Reciprocal Guidance: Orchestrating Draft and Verify Budgets for Advancing the Diffusion-AR Self-Speculation Frontiercited by 0
  2. 2026UBTree: Parallel Tree Drafting via Unigram and Bigram Models for Speculative Decodingcited by 0
  3. 2026DRelay: Global Draft Context for Prefix-Aware Parallel Speculative Decoding Repaircited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Speculative Decoding: Performance or Illusion?
  2. 2024SSSD: Simply-Scalable Speculative Decoding
  3. 2026SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding
  4. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding
  5. 2026MineDraft: A Framework for Batch Parallel Speculative Decoding
  6. 2026Speculative Decoding with a Speculative Vocabulary
  7. 2024Closer Look at Efficient Inference Methods: A Survey of Speculative Decoding
  8. 2024Decoding Speculative Decoding