mlsanityData to 7 Oct 2026

EAGLE-Pangu: Accelerator-Safe Tree Speculative Decoding on Ascend NPUs

arXiv 2603.08088 ↗First posted 9 Mar 2026cs.LG cs.PLcited by 0

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

Autoregressive decoding remains a primary bottleneck in large language model (LLM) serving, motivating speculative decoding methods that reduce expensive teacher-model invocations by verifying multiple candidate tokens per step. Tree-structured speculation further increases parallelism, but is often brittle when ported across heterogeneous backends and accelerator stacks, where attention masking, KV-cache layouts, and indexing semantics are not interchangeable. We present EAGLE-Pangu, a reproducible system that ports EAGLE-3-style tree speculative decoding to a Pangu teacher backend on Ascend NPUs. EAGLE-Pangu contributes (i) an explicit branch/commit cache manager built on the Cache API, (ii) accelerator-safe tree tensorization that removes undefined negative indices by construction and validates structural invariants, and (iii) a fused-kernel-compatible teacher verification path with a debuggable eager fallback. On 240 turns from MT-Bench and HumanEval-style prompts, EAGLE-Pangu improves end-to-end decoding throughput by 1.27x on average, up to 2.46x at p99, over teacher-only greedy decoding in the fused-kernel performance path. We also provide a fused-kernel-free reference path with structured traces and invariant checks to support reproducible debugging and ablation across execution modes and tree budgets.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 4 of its 4 references that are in the map. First posted on arXiv 9 Mar 2026.

Builds on

4 of 4 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test3 Mar
  2. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads19 Jan
  3. 2022Fast Inference from Transformers via Speculative Decoding30 Nov

Other references · 1

  1. 2025Pangu Embedded: An Efficient Dual-system LLM Reasoner with Metacognitioncited by 61

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026CATS: Cascaded Adaptive Tree Speculation for Memory-Limited LLM Inference Acceleration
  2. 2026JetSpec: Breaking the Scaling Ceiling of Speculative Decoding with Parallel Tree Drafting
  3. 2026Goose: Anisotropic Speculation Trees for Training-Free Speculative Decoding
  4. 2025STree: Speculative Tree Decoding for Hybrid State-Space Models
  5. 2024Falcon: Faster and Parallel Inference of Large Language Models through Enhanced Semi-Autoregressive Drafting and Custom-Designed Decoding Tree
  6. 2026ECHO: Elastic Speculative Decoding with Sparse Gating for High-Concurrency Scenarios
  7. 2024HADES: Hardware Accelerated Decoding for Efficient Speculation in Large Language Models
  8. 2024Recursive Speculative Decoding: Accelerating LLM Inference via Sampling Without Replacement