# GRIFFIN: Effective Token Alignment for Faster Speculative Decoding

arXiv: https://arxiv.org/abs/2502.11018 · HTML: https://mlsanity.com/paper/2502.11018/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving.
- Builds on 32 of its 37 references that are in the map.
- First posted on arXiv 16 Feb 2025.

## Placed in

- Efficient LLM inference and serving / [Speculative decoding](https://mlsanity.com/inference/speculative-decoding/)

## Builds on (most cited first, 30 of 32)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [Fast Inference from Transformers via Speculative Decoding](https://mlsanity.com/paper/2211.17192/) (2022)
- [Accelerating Large Language Model Decoding with Speculative Sampling](https://mlsanity.com/paper/2302.01318/) (2023)
- [Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads](https://mlsanity.com/paper/2401.10774/) (2024)
- [Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks](https://mlsanity.com/paper/1506.03099/) (2015)
- [EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty](https://mlsanity.com/paper/2401.15077/) (2024)
- [EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees](https://mlsanity.com/paper/2406.16858/) (2024)
- [SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification](https://mlsanity.com/paper/2305.09781/) (2023)
- [Break the Sequential Dependency of LLM Inference Using Lookahead Decoding](https://mlsanity.com/paper/2402.02057/) (2024)
- [DistillSpec: Improving Speculative Decoding via Knowledge Distillation](https://mlsanity.com/paper/2310.08461/) (2023)
- [Speculative Decoding with Big Little Decoder](https://mlsanity.com/paper/2302.07863/) (2023)
- [REST: Retrieval-Based Speculative Decoding](https://mlsanity.com/paper/2311.08252/) (2023)
- [SpecTr: Fast Speculative Decoding via Optimal Transport](https://mlsanity.com/paper/2310.15141/) (2023)
- [Generalization in Generation: A closer look at Exposure Bias](https://mlsanity.com/paper/1910.00292/) (2019)
- [Hydra: Sequentially-Dependent Draft Heads for Medusa Decoding](https://mlsanity.com/paper/2402.05109/) (2024)
- [Online Speculative Decoding](https://mlsanity.com/paper/2310.07177/) (2023)
- [Sequoia: Scalable, Robust, and Hardware-aware Speculative Decoding](https://mlsanity.com/paper/2402.12374/) (2024)
- [Cascade Speculative Drafting for Even Faster LLM Inference](https://mlsanity.com/paper/2312.11462/) (2023)
- [GliDe with a CaPE: A Low-Hassle Method to Accelerate Speculative Decoding](https://mlsanity.com/paper/2402.02082/) (2024)
- [CLLMs: Consistency Large Language Models](https://mlsanity.com/paper/2403.00835/) (2024)
- [Learning Harmonized Representations for Speculative Sampling](https://mlsanity.com/paper/2408.15766/) (2024)
- [SpecExec: Massively Parallel Speculative Decoding for Interactive LLM Inference on Consumer Devices](https://mlsanity.com/paper/2406.02532/) (2024)
- [Recurrent Drafter for Fast Speculative Decoding in Large Language Models](https://mlsanity.com/paper/2403.09919/) (2024)

## Built on by (most cited first, 16 of 16)

- [A Survey on Parallel Text Generation: From Parallel Decoding to Diffusion Language Models](https://mlsanity.com/paper/2508.08712/) (2025)
- [D-PACE: Dynamic Position-Aware Cross-Entropy for Parallel Speculative Drafting](https://mlsanity.com/paper/2605.18810/) (2026)
- [MoE-SpeQ: Speculative Quantized Decoding with Proactive Expert Prefetching and Offloading for Mixture-of-Experts](https://mlsanity.com/paper/2511.14102/) (2025)
- [Bridging Draft Policy Misalignment: Group Tree Optimization for Speculative Decoding](https://mlsanity.com/paper/2509.22134/) (2025)
- [R-Stitch: Dynamic Trajectory Stitching for Efficient Reasoning](https://mlsanity.com/paper/2507.17307/) (2025)
- [Verification-Aware Training for Speculative Decoding](https://mlsanity.com/paper/2608.30135/) (2026)
- [Make Every Draft Count: Hidden State based Speculative Decoding](https://mlsanity.com/paper/2602.21224/) (2026)
- [Spec-AUF: Accept-Until-Fail Training under Train-Inference Misalignment for Masked Block Drafters](https://mlsanity.com/paper/2607.01893/) (2026)
- [POSS: Position Specialist Generates Better Draft for Speculative Decoding](https://mlsanity.com/paper/2506.03566/) (2025)
- [Variational Speculative Decoding: Rethinking Draft Training from Token Likelihood to Sequence Acceptance](https://mlsanity.com/paper/2602.05774/) (2026)
- [SMART: When is it Actually Worth Expanding a Speculative Tree?](https://mlsanity.com/paper/2604.09731/) (2026)
- [Mistletoe: Stealthy Acceleration-Collapse Attacks on Speculative Decoding](https://mlsanity.com/paper/2605.14005/) (2026)
- [ConFu: Contemplate the Future for Better Speculative Sampling](https://mlsanity.com/paper/2603.08899/) (2026)
- [Position-Aware Drafting for Inference Acceleration in LLM-Based Generative List-Wise Recommendation](https://mlsanity.com/paper/2604.27747/) (2026)
- [Performance-Driven Policy Optimization for Speculative Decoding with Adaptive Windowing](https://mlsanity.com/paper/2605.14978/) (2026)
- [From Position Risks to Block Survival: Faster Generation for Diffusion Language Models](https://mlsanity.com/paper/2609.33390/) (2026)

## Abstract (arXiv)

Speculative decoding accelerates inference in large language models (LLMs) by generating multiple draft tokens simultaneously. However, existing methods often struggle with token misalignment between the training and decoding phases, limiting their performance. To address this, we propose GRIFFIN, a novel framework that incorporates a token-alignable training strategy and a token-alignable draft model to mitigate misalignment. The training strategy employs a loss masking mechanism to exclude highly misaligned tokens during training, preventing them from negatively impacting the draft model's optimization. The token-alignable draft model introduces input tokens to correct inconsistencies in generated features. Experiments on LLaMA, Vicuna, Qwen and Mixtral models demonstrate that GRIFFIN achieves an average acceptance length improvement of over 8% and a speedup ratio exceeding 7%, outperforming current speculative decoding state-of-the-art methods. Our code and GRIFFIN's draft models are released publicly in https://github.com/hsj576/GRIFFIN.