Data to 14 Oct 2026

SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Acceleration

arXiv 2410.06916 ↗First posted 9 Oct 2024cs.CLcited by 72

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

Speculative decoding (SD) has emerged as a widely used paradigm to accelerate LLM inference without compromising quality. It works by first employing a compact model to draft multiple tokens efficiently and then using the target LLM to verify them in parallel. While this technique has achieved notable speedups, most existing approaches necessitate either additional parameters or extensive training to construct effective draft models, thereby restricting their applicability across different LLMs and tasks. To address this limitation, we explore a novel plug-and-play SD solution with layer-skipping, which skips intermediate layers of the target LLM as the compact draft model. Our analysis reveals that LLMs exhibit great potential for self-acceleration through layer sparsity and the task-specific nature of this sparsity. Building on these insights, we introduce SWIFT, an on-the-fly self-speculative decoding algorithm that adaptively selects intermediate layers of LLMs to skip during inference. SWIFT does not require auxiliary models or additional training, making it a plug-and-play solution for accelerating LLM inference across diverse input data streams. Our extensive experiments across a wide range of models and downstream tasks demonstrate that SWIFT can achieve over a 1.3x-1.6x speedup while preserving the original distribution of the generated text. We release our code in https://github.com/hemingkx/SWIFT.

Key facts. Cited by 72 papers in the map (data through 14 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 47 of its 54 references that are in the map. First posted on arXiv 9 Oct 2024.

Builds on

47 of 54 references.

Closest ancestors · 23

Same direction, newest first

11 more

Other references · 24

2023GPT-4 Technical Reportcited by 18,396
2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
2021Training Verifiers to Solve Math Word Problemscited by 10,109
2021Evaluating Large Language Models Trained on Codecited by 8,320
2020Training data-efficient image transformers & distillation through attentioncited by 4,586
2023Code Llama: Open Foundation Models for Codecited by 2,139
2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,181
2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 812
2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 737
2024Yi: Open Foundation Models by 01.AIcited by 719
2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717

12 more

2018Rethinking the Value of Network Pruningcited by 599
2022Efficiently Scaling Transformer Inferencecited by 539
2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 527
2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 449
2024Better & Faster Large Language Models via Multi-token Predictioncited by 360
2018Blockwise Parallel Decoding for Deep Autoregressive Modelscited by 323
2024The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bitscited by 308
2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decodingcited by 307
2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 252
2023SPEED: Speculative Pipelined Execution for Efficient Decodingcited by 47
2023Predictive Pipelined Decoding: A Compute-Latency Trade-off for Exact LLM Decodingcited by 40
2024Accelerating Inference in Large Language Models with a Unified Layer Skipping Strategycited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 32; 2026: 39.

72 papers.

Sort:

By area:

2025A Survey on Parallel Text Generation: From Parallel Decoding to Diffusion Language Modelscited by 30
2025PARD: Accelerating LLM Inference with Low-Cost PARallel Draft Model Adaptationcited by 29
2025Speculative Decoding and Beyond: An In-Depth Survey of Techniquescited by 23
2025ViSpec: Accelerating Vision-Language Models with Vision-Aware Speculative Decodingcited by 22
2025LongSpec: Long-Context Lossless Speculative Decoding with Efficient Drafting and Verificationcited by 21
2025SpecVLM: Enhancing Speculative Decoding of Video LLMs via Verifier-Guided Token Pruningcited by 20
2026SpecForge: A Flexible and Efficient Open-Source Training Framework for Speculative Decodingcited by 16
2025DREAM: Drafting with Refined Target Features and Entropy-Adaptive Cross-Attention Fusion for Multimodal Speculative Decodingcited by 14
2025LogitSpec: Accelerating Retrieval-based Speculative Decoding via Next Next Token Speculationcited by 14
2025KNN-SSD: Enabling Dynamic Self-Speculative Decoding via Nearest Neighbor Layer Set Optimizationcited by 11
2025CLaSp: In-Context Layer Skip for Self-Speculative Decodingcited by 11
2025A Survey on Parallel Reasoningcited by 11

38 more

50 most cited of 72 All 72 (JSON)

Similar, not cited

Close in wording, no citation either way