Acceleration Multiple Heads Decoding for LLM via Dynamic Tree Attention
Not on the map. Why?
Abstract (arXiv)
Multiple heads decoding accelerates the inference of Large Language Models (LLMs) by predicting next several tokens simultaneously. It generates and verifies multiple candidate sequences in parallel via tree attention with a fixed structure. In this paper, we replace the fixed tree attention with dynamic tree attention on multiple head decoding, specifically in the context of MEDUSA. We propose a simple and low complexity strategy to generate candidates and construct the dynamic tree structure. Preliminary experiments show that the proposed method improves the decoding efficiency of multiple head decoding for LLMs while maintaining the generation quality. This result demonstrates the potential for improvement of multiple head decoding in candidate generation.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 9 of its 13 references that are in the map. First posted on arXiv 9 Feb 2025.
Builds on
9 of 13 references.
References · 9
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
- 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
- 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 1,006
- 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 843
- 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
- 2018Blockwise Parallel Decoding for Deep Autoregressive Modelscited by 323
- 2023Accelerating Transformer Inference for Translation via Parallel Decodingcited by 124
- 2020Accelerating Feedforward Computation via Parallel Nonlinear Equation Solvingcited by 52
Built on by
Not cited yet.