# Annealed Relaxation of Speculative Decoding for Faster Autoregressive Image Generation

arXiv: https://arxiv.org/abs/2601.09212 · HTML: https://mlsanity.com/paper/2601.09212/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Fast autoregressive visual generation, one of 12 directions in Efficient LLM inference and serving.
- Builds on 23 of its 36 references that are in the map.
- First posted on arXiv 14 Jan 2026.

## Placed in

- Efficient LLM inference and serving / [Fast autoregressive visual generation](https://mlsanity.com/inference/fast-autoregressive-visual-generation/)

## Builds on (most cited first, 23 of 23)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [Fast Inference from Transformers via Speculative Decoding](https://mlsanity.com/paper/2211.17192/) (2022)
- [Accelerating Large Language Model Decoding with Speculative Sampling](https://mlsanity.com/paper/2302.01318/) (2023)
- [Chameleon: Mixed-Modal Early-Fusion Foundation Models](https://mlsanity.com/paper/2405.09818/) (2024)
- [Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads](https://mlsanity.com/paper/2401.10774/) (2024)
- [Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling](https://mlsanity.com/paper/2501.17811/) (2025)
- [Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation](https://mlsanity.com/paper/2406.06525/) (2024)
- [EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees](https://mlsanity.com/paper/2406.16858/) (2024)
- [Lumina-mGPT: Illuminate Flexible Photorealistic Text-to-Image Generation with Multimodal Generative Pretraining](https://mlsanity.com/paper/2408.02657/) (2024)
- [GliDe with a CaPE: A Low-Hassle Method to Accelerate Speculative Decoding](https://mlsanity.com/paper/2402.02082/) (2024)
- [Accelerating Auto-regressive Text-to-Image Generation with Training-free Speculative Jacobi Decoding](https://mlsanity.com/paper/2410.01699/) (2024)
- [LANTERN: Accelerating Visual Autoregressive Models with Relaxed Speculative Decoding](https://mlsanity.com/paper/2410.03355/) (2024)
- [A Theoretical Perspective for Speculative Decoding Algorithm](https://mlsanity.com/paper/2411.00841/) (2024)
- [Next Patch Prediction for Autoregressive Visual Generation](https://mlsanity.com/paper/2412.15321/) (2024)
- [ZipAR: Parallel Auto-regressive Image Generation through Spatial Locality](https://mlsanity.com/paper/2412.04062/) (2024)
- [Speeding up Speculative Decoding via Sequential Approximate Verification](https://mlsanity.com/paper/2502.04557/) (2025)
- [LANTERN++: Enhancing Relaxed Speculative Decoding with Static Tree Drafting for Visual Auto-regressive Models](https://mlsanity.com/paper/2502.06352/) (2025)

## Built on by (most cited first, 1 of 1)

- [SSD: Spatially Speculative Decoding Accelerates Autoregressive Image Generation](https://mlsanity.com/paper/2606.20543/) (2026)

## Abstract (arXiv)

Despite significant progress in autoregressive image generation, inference remains slow due to the sequential nature of AR models and the ambiguity of image tokens, even when using speculative decoding. Recent works attempt to address this with relaxed speculative decoding but lack theoretical grounding. In this paper, we establish the theoretical basis of relaxed SD and propose COOL-SD, an annealed relaxation of speculative decoding built on two key insights. The first analyzes the total variation (TV) distance between the target model and relaxed speculative decoding and yields an optimal resampling distribution that minimizes an upper bound of the distance. The second uses perturbation analysis to reveal an annealing behaviour in relaxed speculative decoding, motivating our annealed design. Together, these insights enable COOL-SD to generate images faster with comparable quality, or achieve better quality at similar latency. Experiments validate the effectiveness of COOL-SD, showing consistent improvements over prior methods in speed-quality trade-offs.