mlsanityData to 14 Oct 2026

Compiler-Assisted Speculative Sampling for Accelerated LLM Inference on Heterogeneous Edge Devices

arXiv 2602.08060 ↗First posted 8 Feb 2026cs.LGcited by 0

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

LLM deployment on resource-constrained edge devices faces severe latency constraints, particularly in real-time applications where delayed responses can compromise safety or usability. Among many approaches to mitigate the inefficiencies of sequential token-by-token generation, Speculative Decoding (SD) has emerged as a promising technique. However, SD at the edge is hindered by two major challenges: (1) integrating SD into a compiler-based workflow without sacrificing performance or programmability, and (2) exploiting the heterogeneous compute resources of modern SoCs through carefully designed partitioning strategies. This work addresses these challenges by using an analytical cost model that explores heterogeneous hardware configurations and guides coarse-grained partitioning of LLM subgraphs, particularly with edge-typical short input sequence lengths. The cost model predicts when speculative sampling and heterogeneous execution are jointly beneficial and is validated on an edge device featuring a hexacore Cortex-A CPU and a Mali GPU, revealing up to 1.68× speedup for translation tasks, closely matching analytic expectations.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 14 of its 24 references that are in the map. First posted on arXiv 8 Feb 2026.

Builds on

14 of 24 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025Speculative Decoding Meets Quantization: Compatibility Evaluation and Hierarchical Framework Design28 May
  2. 2025DuoDecoding: Hardware-aware Heterogeneous Speculative Decoding with Dynamic Multi-Sequence Drafting2 Mar
  3. 2025Speculative Decoding and Beyond: An In-Depth Survey of Techniques27 Feb
  4. 2024Dovetail: A CPU/GPU Heterogeneous Speculative Decoding for LLM inference25 Dec
  5. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty26 Jan
  6. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding15 Jan
  7. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification16 May
  8. 2022Fast Inference from Transformers via Speculative Decoding30 Nov

Other references · 6

  1. 2024A Review on Edge Large Language Models: Design, Execution, and Applicationscited by 100
  2. 2023Full Stack Optimization of Transformer Inference: a Surveycited by 80
  3. 2018Glow: Graph Lowering Compiler Techniques for Neural Networkscited by 75
  4. 2018Tiramisu: A Polyhedral Compiler for Expressing Fast and Portable Codecited by 53
  5. 2020The Deep Learning Compiler: A Comprehensive Surveycited by 46
  6. 2021MAGMA: An Optimization Framework for Mapping Multiple DNNs on Multiple Accelerator Corescited by 15

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025SLED: A Speculative LLM Decoding Framework for Efficient Edge Serving
  2. 2025Mirror Speculative Decoding: Breaking the Serial Barrier in LLM Inference
  3. 2025Yggdrasil: Bridging Dynamic Speculation and Static Runtime for Latency-Optimal Tree-Based LLM Decoding
  4. 2025Ghidorah: Fast LLM Inference on Edge with Speculative Decoding and Hetero-Core Parallelism
  5. 2024SpecExec: Massively Parallel Speculative Decoding for Interactive LLM Inference on Consumer Devices
  6. 2026WISP: Waste- and Interference-Suppressed Distributed Speculative LLM Serving at the Edge via Dynamic Drafting and SLO-Aware Batching
  7. 2025Scaling LLM Speculative Decoding: Non-Autoregressive Forecasting in Large-Batch Scenarios
  8. 2026BitNest: Bit-Nested Speculative Decoding for Memory-Efficient LLM Inference Acceleration