mlsanityData to 7 Oct 2026

Ghidorah: Fast LLM Inference on Edge with Speculative Decoding and Hetero-Core Parallelism

arXiv 2505.23219 ↗First posted 29 May 2025cs.DCcited by 1

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

In-situ LLM inference on end-user devices has gained significant interest due to its privacy benefits and reduced dependency on external infrastructure. However, as the decoding process is memory-bandwidth-bound, the diverse processing units in modern end-user devices cannot be fully exploited, resulting in slow LLM inference. This paper presents Ghidorah, a LLM inference system for end-user devices with the unified memory architecture. The key idea of Ghidorah can be summarized in two steps: 1) leveraging speculative decoding approaches to enhance parallelism, and 2) ingeniously distributing workloads across multiple heterogeneous processing units to maximize computing power utilization. Ghidorah includes the hetero-core model parallelism (HCMP) architecture and the architecture-aware profiling (ARCA) approach. The HCMP architecture guides partitioning by leveraging the unified memory design of end-user devices and adapting to the hybrid computational demands of speculative decoding. The ARCA approach is used to determine the optimal speculative strategy and partitioning strategy, balancing acceptance rate with parallel capability to maximize the speedup. Additionally, we optimize sparse computation on ARM CPUs. Experimental results show that Ghidorah can achieve up to 7.6x speedup in the dominant LLM decoding phase compared to the sequential decoding approach in NVIDIA Jetson NX.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 22 of its 36 references that are in the map. First posted on arXiv 29 May 2025.

Builds on

22 of 36 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Sequoia: Scalable, Robust, and Hardware-aware Speculative Decoding19 Feb
  2. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decoding3 Feb
  3. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty26 Jan
  4. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads19 Jan
  5. 2023DistillSpec: Improving Speculative Decoding via Knowledge Distillation12 Oct
  6. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification16 May

Other references · 16

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  7. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  8. 2021Program Synthesis with Large Language Modelscited by 3,509
  9. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  10. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 507
  11. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 448
  12. 2023DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Modelscited by 239
4 more
  1. 2022Petals: Collaborative Inference and Fine-tuning of Large Modelscited by 84
  2. 2023LLMCad: Fast and Scalable On-device Large Language Model Inferencecited by 45
  3. 2024Galaxy: A Resource-Efficient Collaborative Edge AI System for In-situ Transformer Inferencecited by 25
  4. 2019High-Throughput CNN Inference on Embedded ARM big.LITTLE Multi-Core Processorscited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026SPECTRA: Adaptive Execution of Speculative Decoding on a Runtime-Reconfigurable Tiled Architecturecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Compiler-Assisted Speculative Sampling for Accelerated LLM Inference on Heterogeneous Edge Devices
  2. 2025Mirror Speculative Decoding: Breaking the Serial Barrier in LLM Inference
  3. 2025SLED: A Speculative LLM Decoding Framework for Efficient Edge Serving
  4. 2024Dovetail: A CPU/GPU Heterogeneous Speculative Decoding for LLM inference
  5. 2026BALANCE: Hybrid Autoregressive-Speculative LLM Inference at the Network Edge
  6. 2025Yggdrasil: Bridging Dynamic Speculation and Static Runtime for Latency-Optimal Tree-Based LLM Decoding
  7. 2025CARD: A Cache-Assisted Parallel Speculative Decoding Framework via Query-and-Correct Paradigm for Accelerating LLM Inference
  8. 2025Scaling LLM Speculative Decoding: Non-Autoregressive Forecasting in Large-Batch Scenarios