# Falcon: Faster and Parallel Inference of Large Language Models through Enhanced Semi-Autoregressive Drafting and Custom-Designed Decoding Tree

arXiv: https://arxiv.org/abs/2412.12639 · HTML: https://mlsanity.com/paper/2412.12639/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving.
- Builds on 21 of its 27 references that are in the map.
- First posted on arXiv 17 Dec 2024.

## Placed in

- Efficient LLM inference and serving / [Speculative decoding](https://mlsanity.com/inference/speculative-decoding/)

## Builds on (most cited first, 21 of 21)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [GLM: General Language Model Pretraining with Autoregressive Blank Infilling](https://mlsanity.com/paper/2103.10360/) (2021)
- [Accelerating Large Language Model Decoding with Speculative Sampling](https://mlsanity.com/paper/2302.01318/) (2023)
- [Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads](https://mlsanity.com/paper/2401.10774/) (2024)
- [EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty](https://mlsanity.com/paper/2401.15077/) (2024)
- [Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding](https://mlsanity.com/paper/2401.07851/) (2024)
- [Speculative Decoding with Big Little Decoder](https://mlsanity.com/paper/2302.07863/) (2023)
- [Fast Decoding in Sequence Models using Discrete Latent Variables](https://mlsanity.com/paper/1803.03382/) (2018)
- [Cascade Speculative Drafting for Even Faster LLM Inference](https://mlsanity.com/paper/2312.11462/) (2023)
- [A Survey on Non-Autoregressive Generation for Neural Machine Translation and Beyond](https://mlsanity.com/paper/2204.09269/) (2022)
- [Fast Structured Decoding for Sequence Models](https://mlsanity.com/paper/1910.11555/) (2019)
- [Understanding and Improving Lexical Choice in Non-Autoregressive Translation](https://mlsanity.com/paper/2012.14583/) (2020)
- [Non-Autoregressive Machine Translation with Auxiliary Regularization](https://mlsanity.com/paper/1902.10245/) (2019)
- [Semi-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/1808.08583/) (2018)
- [Fine-Tuning by Curriculum Learning for Non-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/1911.08717/) (2019)
- [How Does Distilled Data Complexity Impact the Quality and Confidence of Non-Autoregressive Machine Translation?](https://mlsanity.com/paper/2105.12900/) (2021)
- [Self-Distillation Mixup Training for Non-autoregressive Neural Machine Translation](https://mlsanity.com/paper/2112.11640/) (2021)

## Built on by (most cited first, 20 of 20)

- [EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test](https://mlsanity.com/paper/2503.01840/) (2025)
- [A Survey on Parallel Text Generation: From Parallel Decoding to Diffusion Language Models](https://mlsanity.com/paper/2508.08712/) (2025)
- [DART: Diffusion-Inspired Speculative Decoding for Fast LLM Inference](https://mlsanity.com/paper/2601.19278/) (2026)
- [Speculative Decoding and Beyond: An In-Depth Survey of Techniques](https://mlsanity.com/paper/2502.19732/) (2025)
- [TALON: Confidence-Aware Speculative Decoding with Adaptive Token Trees](https://mlsanity.com/paper/2601.07353/) (2026)
- [PARD-2: Target-Aligned Parallel Draft Model for Dual-Mode Speculative Decoding](https://mlsanity.com/paper/2605.08632/) (2026)
- [RASD: Retrieval-Augmented Speculative Decoding](https://mlsanity.com/paper/2503.03434/) (2025)
- [P-EAGLE: Parallel-Drafting EAGLE with Scalable Training](https://mlsanity.com/paper/2602.01469/) (2026)
- [Scaling LLM Speculative Decoding: Non-Autoregressive Forecasting in Large-Batch Scenarios](https://mlsanity.com/paper/2511.20340/) (2025)
- [From Static Inference to Dynamic Interaction: A Survey of Streaming Large Language Models](https://mlsanity.com/paper/2603.04592/) (2026)
- [SpecBlock: Block-Iterative Speculative Decoding with Dynamic Tree Drafting](https://mlsanity.com/paper/2605.07243/) (2026)
- [Feather-SQL: A Lightweight NL2SQL Framework with Dual-Model Collaboration Paradigm for Small Language Models](https://mlsanity.com/paper/2503.17811/) (2025)
- [SpecFLASH: A Latent-Guided Semi-autoregressive Speculative Decoding Framework for Efficient Multimodal Generation](https://mlsanity.com/paper/2505.12728/) (2025)
- [Polybasic Speculative Decoding Through a Theoretical Perspective](https://mlsanity.com/paper/2510.26527/) (2025)
- [Margins, Not Windows: Training-Free Per-Step Lossy Speculative Decoding](https://mlsanity.com/paper/2609.02897/) (2026)
- [RADAR: Accelerate Large Language Model Inference With RL-Based Dynamic Draft Trees](https://mlsanity.com/paper/2512.14069/) (2025)
- [Draft-and-Target Sampling for Video Generation Policy](https://mlsanity.com/paper/2603.13438/) (2026)
- [DREAM-R: Multimodal Speculative Reasoning with RL-Based Refined Drafting, Precise Verification, and Fully Parallel Execution](https://mlsanity.com/paper/2605.28678/) (2026)
- [Osprey: Target-agnostic Pre-training Makes Stronger Drafters in Speculative Decoding](https://mlsanity.com/paper/2609.09338/) (2026)
- [Recovering Off-Policy Supervision for Speculative Decoding](https://mlsanity.com/paper/2609.38795/) (2026)

## Abstract (arXiv)

Striking an optimal balance between minimal drafting latency and high speculation accuracy to enhance the inference speed of Large Language Models remains a significant challenge in speculative decoding. In this paper, we introduce Falcon, an innovative semi-autoregressive speculative decoding framework fashioned to augment both the drafter's parallelism and output quality. Falcon incorporates the Coupled Sequential Glancing Distillation technique, which fortifies inter-token dependencies within the same block, leading to increased speculation accuracy. We offer a comprehensive theoretical analysis to illuminate the underlying mechanisms. Additionally, we introduce a Custom-Designed Decoding Tree, which permits the drafter to generate multiple tokens in a single forward pass and accommodates multiple forward passes as needed, thereby boosting the number of drafted tokens and significantly improving the overall acceptance rate. Comprehensive evaluations on benchmark datasets such as MT-Bench, HumanEval, and GSM8K demonstrate Falcon's superior acceleration capabilities. The framework achieves a lossless speedup ratio ranging from 2.91x to 3.51x when tested on the Vicuna and LLaMA2-Chat model series. These results outstrip existing speculative decoding methods for LLMs, including Eagle, Medusa, Lookahead, SPS, and PLD, while maintaining a compact drafter architecture equivalent to merely two Transformer layers.