# D-PACE: Dynamic Position-Aware Cross-Entropy for Parallel Speculative Drafting

arXiv: https://arxiv.org/abs/2605.18810 · HTML: https://mlsanity.com/paper/2605.18810/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving.
- Builds on 40 of its 46 references that are in the map.
- First posted on arXiv 12 May 2026.

## Placed in

- Efficient LLM inference and serving / [Speculative decoding](https://mlsanity.com/inference/speculative-decoding/)

## Builds on (most cited first, 30 of 40)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [Fast Inference from Transformers via Speculative Decoding](https://mlsanity.com/paper/2211.17192/) (2022)
- [Accelerating Large Language Model Decoding with Speculative Sampling](https://mlsanity.com/paper/2302.01318/) (2023)
- [Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads](https://mlsanity.com/paper/2401.10774/) (2024)
- [EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty](https://mlsanity.com/paper/2401.15077/) (2024)
- [Dream 7B: Diffusion Large Language Models](https://mlsanity.com/paper/2508.15487/) (2025)
- [Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models](https://mlsanity.com/paper/2503.09573/) (2025)
- [EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees](https://mlsanity.com/paper/2406.16858/) (2024)
- [EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test](https://mlsanity.com/paper/2503.01840/) (2025)
- [Blockwise Parallel Decoding for Deep Autoregressive Models](https://mlsanity.com/paper/1811.03115/) (2018)
- [Break the Sequential Dependency of LLM Inference Using Lookahead Decoding](https://mlsanity.com/paper/2402.02057/) (2024)
- [Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding](https://mlsanity.com/paper/2401.07851/) (2024)
- [Mercury: Ultra-Fast Language Models Based on Diffusion](https://mlsanity.com/paper/2506.17298/) (2025)
- [DistillSpec: Improving Speculative Decoding via Knowledge Distillation](https://mlsanity.com/paper/2310.08461/) (2023)
- [DFlash: Block Diffusion for Flash Speculative Decoding](https://mlsanity.com/paper/2602.06036/) (2026)
- [Hydra: Sequentially-Dependent Draft Heads for Medusa Decoding](https://mlsanity.com/paper/2402.05109/) (2024)
- [Online Speculative Decoding](https://mlsanity.com/paper/2310.07177/) (2023)
- [GliDe with a CaPE: A Low-Hassle Method to Accelerate Speculative Decoding](https://mlsanity.com/paper/2402.02082/) (2024)
- [Learning Harmonized Representations for Speculative Sampling](https://mlsanity.com/paper/2408.15766/) (2024)
- [OPT-Tree: Speculative Decoding with Adaptive Draft Tree Structure](https://mlsanity.com/paper/2406.17276/) (2024)

## Built on by (most cited first, 17 of 17)

- [Breaking Entropy Bounds: Accelerating RL Training via MTP with Rejection Sampling](https://mlsanity.com/paper/2606.12370/) (2026)
- [Verification-Aware Training for Speculative Decoding](https://mlsanity.com/paper/2608.30135/) (2026)
- [DeLS-Spec: Decoupled Long-Short Contexts for Parallel Speculative Drafting](https://mlsanity.com/paper/2607.07409/) (2026)
- [AngelSpec: Towards Real-World High Performance Inference with Speculative Decoding](https://mlsanity.com/paper/2607.25852/) (2026)
- [LiLiCorr: Lightweight Likelihood Correlation of Parallel Drafts for Speculative Decoding](https://mlsanity.com/paper/2608.20530/) (2026)
- [Spec-AUF: Accept-Until-Fail Training under Train-Inference Misalignment for Masked Block Drafters](https://mlsanity.com/paper/2607.01893/) (2026)
- [DFlow: Enabling Verifier Information Flow in Block Diffusion Speculative Decoding](https://mlsanity.com/paper/2609.06498/) (2026)
- [Beyond Parallel Blindness: Information Floors and Model Gaps in Block Drafting](https://mlsanity.com/paper/2608.27339/) (2026)
- [Carryover Drafting: Recycling Rejected States for Speculative Decoding](https://mlsanity.com/paper/2609.14717/) (2026)
- [Draft in Parallel, Condition Through Depth: Adjacent Causal Injection for Speculative Decoding](https://mlsanity.com/paper/2609.36173/) (2026)
- [Ceiling-Clipped Acceptance Histograms Indicate Stranded Speed-up in Block-Diffusion Speculative Decoding](https://mlsanity.com/paper/2608.30427/) (2026)
- [Vision Is Not Overhead: One-Pass Block Drafting for Lossless Speculative Decoding in Vision-Language Models](https://mlsanity.com/paper/2609.00355/) (2026)
- [Diffusion Drafts, AR Verifies: Accelerating Document OCR with Self-Speculative Decoding](https://mlsanity.com/paper/2609.26638/) (2026)
- [From Position Risks to Block Survival: Faster Generation for Diffusion Language Models](https://mlsanity.com/paper/2609.33390/) (2026)
- [DRelay: Global Draft Context for Prefix-Aware Parallel Speculative Decoding Repair](https://mlsanity.com/paper/2610.01439/) (2026)
- [AdaSpark: Adaptive DSpark with Online Learning for Tree Verification and N-gram Fill](https://mlsanity.com/paper/2610.05774/) (2026)
- [Training Parallel Speculative Draft Models by Directly Minimizing Expected Decoding Rounds](https://mlsanity.com/paper/2610.10411/) (2026)

## Abstract (arXiv)

Speculative decoding accelerates LLM inference by having a small drafter propose tokens that a larger target model verifies in parallel. Recent diffusion-based parallel drafters such as DFlash predict the full B-token block in one forward pass, enabling deeper drafters and longer accepted blocks. However, existing multi-token drafter objectives often use fixed position-dependent weighting schedules, such as head-dependent weights or block-position decays, which do not adapt as the positions limiting acceptance change during training. To address this, we derive per-position training weights from a differentiable surrogate of expected accepted draft length, matching the weight of each position to its log-probability gradient contribution. The resulting loss, D-PACE (Dynamic Position-Aware Cross-Entropy), shifts training signal toward positions that currently limit acceptance as the drafter improves. Across six benchmarks, two Qwen3-4B draft depths, two decoding temperatures, and two additional target models, D-PACE consistently improves both wall-clock speedup and average emitted length, with 2.3% measured training-time overhead and no changes to the drafter architecture or inference procedure.