# Nightjar: Dynamic Adaptive Speculative Decoding for Large Language Models Serving

arXiv: https://arxiv.org/abs/2512.22420 · HTML: https://mlsanity.com/paper/2512.22420/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving.
- Builds on 21 of its 47 references that are in the map.
- First posted on arXiv 27 Dec 2025.

## Placed in

- Efficient LLM inference and serving / [Speculative decoding](https://mlsanity.com/inference/speculative-decoding/)

## Builds on (most cited first, 21 of 21)

- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Fast Inference from Transformers via Speculative Decoding](https://mlsanity.com/paper/2211.17192/) (2022)
- [Efficiently Scaling Transformer Inference](https://mlsanity.com/paper/2211.05102/) (2022)
- [EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees](https://mlsanity.com/paper/2406.16858/) (2024)
- [Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding](https://mlsanity.com/paper/2401.07851/) (2024)
- [Batched Multi-armed Bandits Problem](https://mlsanity.com/paper/1904.01763/) (2019)
- [MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decoding](https://mlsanity.com/paper/2408.11049/) (2024)
- [SpecDec++: Boosting Speculative Decoding via Adaptive Candidate Lengths](https://mlsanity.com/paper/2405.19715/) (2024)
- [OPT-Tree: Speculative Decoding with Adaptive Draft Tree Structure](https://mlsanity.com/paper/2406.17276/) (2024)
- [Efficient Contextual Bandits in Non-stationary Worlds](https://mlsanity.com/paper/1708.01799/) (2017)
- [A Survey of Techniques for Optimizing Transformer Inference](https://mlsanity.com/paper/2307.07982/) (2023)
- [70% Size, 100% Accuracy: Lossless LLM Compression for Efficient GPU Inference via Dynamic-Length Float (DFloat11)](https://mlsanity.com/paper/2504.11651/) (2025)
- [Dynamic Depth Decoding: Faster Speculative Decoding for LLMs](https://mlsanity.com/paper/2409.00142/) (2024)
- [BanditSpec: Adaptive Speculative Decoding via Bandit Algorithms](https://mlsanity.com/paper/2505.15141/) (2025)
- [The Synergy of Speculative Decoding and Batching in Serving Large Language Models](https://mlsanity.com/paper/2310.18813/) (2023)
- [AdaEAGLE: Optimizing Speculative Decoding via Explicit Modeling of Adaptive Draft Structures](https://mlsanity.com/paper/2412.18910/) (2024)
- [TETRIS: Optimal Draft Token Selection for Batch Speculative Decoding](https://mlsanity.com/paper/2502.15197/) (2025)
- [Dovetail: A CPU/GPU Heterogeneous Speculative Decoding for LLM inference](https://mlsanity.com/paper/2412.18934/) (2024)
- [eLLM: Elastic Memory Management Framework for Efficient LLM Serving](https://mlsanity.com/paper/2506.15155/) (2025)
- [FineServe: Precision-Aware KV Slab and Two-Level Scheduling for Heterogeneous Precision LLM Serving](https://mlsanity.com/paper/2509.06261/) (2025)
- [Adaptive Endpointing with Deep Contextual Multi-armed Bandits](https://mlsanity.com/paper/2303.13407/) (2023)

## Built on by (most cited first, 6 of 6)

- [FASER: Fine-Grained Phase Management for Speculative Decoding in Dynamic LLM Serving](https://mlsanity.com/paper/2604.20503/) (2026)
- [Regulating Branch Parallelism in LLM Serving](https://mlsanity.com/paper/2605.06914/) (2026)
- [Hybrid Verified Decoding: Learning to Allocate Verification in Speculative Decoding](https://mlsanity.com/paper/2606.01019/) (2026)
- [Fully Rotation-Equivariant Spectral-Spatial Learning for Multispectral Object Detection](https://mlsanity.com/paper/2607.05148/) (2026)
- [DynBranch: Speculative Subgraph Reuse for Dynamic Agentic LLM Serving](https://mlsanity.com/paper/2609.31047/) (2026)
- [Match the Distribution, Not the Compute: Post-Training Multi-Token Prediction Heads](https://mlsanity.com/paper/2610.00888/) (2026)

## Abstract (arXiv)

Speculative decoding (SD) accelerates LLM inference by verifying draft tokens in parallel. However, this method presents a critical trade-off: it improves throughput in low-load, memory-bound systems but degrades performance in high-load, compute-bound environments due to verification overhead. Existing speculative decoding methods use fixed lengths and cannot adapt to workload changes or decide when to stop speculation. The cost of restarting speculative inference also remains unquantified. Under high load, the benefit of speculation diminishes, while retaining the draft model reduces KV cache capacity, limiting batch size and degrading throughput. To overcome this, we propose Nightjar, a resource-aware adaptive speculative framework. It first adjusts to the request load by dynamically selecting the optimal speculative length for different batch sizes. Crucially, Nightjar proactively disables speculative decoding when the MAB planner determines that speculation is no longer beneficial, and during the disabled phase, offloads the draft model to the CPU only under GPU memory pressure. This reclaims memory for the KV cache, thereby facilitating larger batch sizes and maximizing overall system throughput. Experiments show that Nightjar achieves up to 14.76% higher throughput than standard speculative decoding and up to 20.18% lower latency in the main benchmark suite under dynamic request arrival rates for real-time LLM serving scenarios.