# ASPD: Unlocking Adaptive Serial-Parallel Decoding by Exploring Intrinsic Parallelism in LLMs

arXiv: https://arxiv.org/abs/2508.08895 · HTML: https://mlsanity.com/paper/2508.08895/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving.
- Builds on 27 of its 35 references that are in the map.
- First posted on arXiv 12 Aug 2025.

## Placed in

- Efficient LLM inference and serving / [Speculative decoding](https://mlsanity.com/inference/speculative-decoding/)

## Builds on (most cited first, 27 of 27)

- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [Qwen2 Technical Report](https://mlsanity.com/paper/2407.10671/) (2024)
- [Fast Inference from Transformers via Speculative Decoding](https://mlsanity.com/paper/2211.17192/) (2022)
- [SGLang: Efficient Execution of Structured Language Model Programs](https://mlsanity.com/paper/2312.07104/) (2023)
- [Large Language Monkeys: Scaling Inference Compute with Repeated Sampling](https://mlsanity.com/paper/2407.21787/) (2024)
- [EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty](https://mlsanity.com/paper/2401.15077/) (2024)
- [EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees](https://mlsanity.com/paper/2406.16858/) (2024)
- [Accessing GPT-4 level Mathematical Olympiad Solutions via Monte Carlo Tree Self-refine with LLaMa-3 8B](https://mlsanity.com/paper/2406.07394/) (2024)
- [Sequence-to-Sequence Learning as Beam-Search Optimization](https://mlsanity.com/paper/1606.02960/) (2016)
- [Skeleton-of-Thought: Prompting LLMs for Efficient Parallel Generation](https://mlsanity.com/paper/2307.15337/) (2023)
- [Learning Harmonized Representations for Speculative Sampling](https://mlsanity.com/paper/2408.15766/) (2024)
- [Learning Adaptive Parallel Reasoning with Language Models](https://mlsanity.com/paper/2504.15466/) (2025)
- [Hogwild! Inference: Parallel LLM Generation via Concurrent Attention](https://mlsanity.com/paper/2504.06261/) (2025)
- [Parallel Scaling Law for Language Models](https://mlsanity.com/paper/2505.10475/) (2025)
- [Multiverse: Your Language Models Secretly Decide How to Parallelize and Merge Generation](https://mlsanity.com/paper/2506.09991/) (2025)
- [Learning to Keep a Promise: Scaling Language Model Decoding Parallelism with Learned Asynchronous Decoding](https://mlsanity.com/paper/2502.11517/) (2025)
- [APAR: LLMs Can Do Auto-Parallel Auto-Regressive Decoding](https://mlsanity.com/paper/2401.06761/) (2024)
- [Generation Meets Verification: Accelerating Large Language Model Inference with Smart Parallel Auto-Correct Decoding](https://mlsanity.com/paper/2402.11809/) (2024)
- [Group Think: Multiple Concurrent Reasoning Agents Collaborating at Token Level Granularity](https://mlsanity.com/paper/2505.11107/) (2025)
- [Accelerate Parallelizable Reasoning via Parallel Decoding within One Sequence](https://mlsanity.com/paper/2503.20533/) (2025)

## Built on by (most cited first, 11 of 11)

- [Parallel-R1: Towards Parallel Thinking via Reinforcement Learning](https://mlsanity.com/paper/2509.07980/) (2025)
- [Retrieval-of-Thought: Efficient Reasoning via Reusing Thoughts](https://mlsanity.com/paper/2509.21743/) (2025)
- [A Survey on Parallel Reasoning](https://mlsanity.com/paper/2510.12164/) (2025)
- [The Road Less Traveled: Enhancing Exploration in LLMs via Sequential Sampling](https://mlsanity.com/paper/2510.15502/) (2025)
- [LaneRoPE: Positional Encoding for Collaborative Parallel Reasoning and Generation](https://mlsanity.com/paper/2605.27570/) (2026)
- [Message Passing Enables Efficient Reasoning](https://mlsanity.com/paper/2607.01077/) (2026)
- [PaDoc: Layout-Grounded Parallel Decoding for Document Parsing](https://mlsanity.com/paper/2608.06146/) (2026)
- [Draft in Parallel, Condition Through Depth: Adjacent Causal Injection for Speculative Decoding](https://mlsanity.com/paper/2609.36173/) (2026)
- [Regulating Branch Parallelism in LLM Serving](https://mlsanity.com/paper/2605.06914/) (2026)
- [Fishing Out Free Riders: Shapley-Based Reward Attribution for Parallel Reasoning via Reinforcement Learning](https://mlsanity.com/paper/2607.18979/) (2026)
- [Line-Coupled Language Model](https://mlsanity.com/paper/2609.07129/) (2026)

## Abstract (arXiv)

The increasing scale and complexity of large language models (LLMs) pose significant inference latency challenges, primarily due to their autoregressive decoding paradigm characterized by the sequential nature of next-token prediction. By re-examining the outputs of autoregressive models, we observed that some segments exhibit parallelizable structures, which we term intrinsic parallelism. Decoding each parallelizable branch simultaneously (i.e. parallel decoding) can significantly improve the overall inference speed of LLMs. In this paper, we propose an Adaptive Serial-Parallel Decoding (ASPD), which addresses two core challenges: automated construction of parallelizable data and efficient parallel decoding mechanism. More specifically, we introduce a non-invasive pipeline that automatically extracts and validates parallelizable structures from the responses of autoregressive models. To empower efficient adaptive serial-parallel decoding, we implement a Hybrid Decoding Engine which enables seamless transitions between serial and parallel decoding modes while maintaining a reusable KV cache, maximizing computational efficiency. Extensive evaluations across General Tasks, Retrieval-Augmented Generation, Mathematical Reasoning, demonstrate that ASPD achieves unprecedented performance in both effectiveness and efficiency. Notably, on Vicuna Bench, our method achieves up to 3.19x speedup (1.85x on average) while maintaining response quality within 1% difference compared to autoregressive models, realizing significant acceleration without compromising generation quality. Our framework sets a groundbreaking benchmark for efficient LLM parallel inference, paving the way for its deployment in latency-sensitive applications such as AI-powered customer service bots and answer retrieval engines.