mlsanityData to 14 Oct 2026

GELATO: Generative Entropy- and Lyapunov-based Adaptive Token Offloading for Device-Edge Speculative LLM Inference

arXiv 2605.10124 ↗First posted 11 May 2026cs.NI cs.DC cs.ITcited by 2

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

The recent growth of on-device Large Language Model (LLM) inference has driven significant interest in device-edge collaborative LLM inference. As a promising architecture, Speculative Decoding (SD) is increasingly adopted where a lightweight draft model rapidly generates candidate tokens to be verified by a powerful target model. However, a fundamental challenge lies in achieving per-token resource scheduling to effectively adapt SD paradigm to resource-constrained edge environment. This paper proposes a Generative Entropy- and Lyapunov-based Adaptive Token Offloading framework, named GELATO, to maximize decoding throughput under energy constraints in a device-edge collaborative SD system. Specifically, an outer drift-plus-penalty loop makes online decisions to establish a reference drafting budget, managing long-term energy-throughput trade-off. Further, a nested entropy-driven generation mechanism executes early exiting to adapt to per-token dynamic generative uncertainty. Theoretical analysis establishes a rigorous performance bound on long-term throughput for GELATO. Extensive evaluations demonstrate that GELATO achieves a globally optimal tradeoff, outperforming state-of-the-art distributed SD architectures by 64.98% in token throughput and reducing energy consumption by 47.47% under resource-constrained environments, while preserving LLM decoding quality.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 6 of its 22 references that are in the map. First posted on arXiv 11 May 2026.

Builds on

6 of 22 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Collaborative Large Language Model Inference via Resource-Aware Parallel Speculative Decoding3 Nov
  2. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb

Other references · 4

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2025The ML.ENERGY Benchmark: Toward Automated Inference Energy Measurement and Optimizationcited by 33
  3. 2025Fast and Cost-effective Speculative Edge-Cloud Decoding with Early Exitscited by 11
  4. 2025Communication-Efficient Hybrid Language Model via Uncertainty-Aware Opportunistic and Compressed Transmissioncited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026From Semantic to Token Communication: The Next Paradigm for Large-Model-Driven 6G Intelligent Connectivitycited by 0
  2. 2026End-to-End Latency-Minimizing and Load-Balanced Request Scheduling for Edge LLM Inference in Agentic AI Servicescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025SLED: A Speculative LLM Decoding Framework for Efficient Edge Serving
  2. 2026Compiler-Assisted Speculative Sampling for Accelerated LLM Inference on Heterogeneous Edge Devices
  3. 2026A Pipelined Collaborative Speculative Decoding Framework for Efficient Edge-Cloud LLM Inference
  4. 2025Speculative Decoding in Decentralized LLM Inference: Turning Communication Latency into Computation Throughput
  5. 2026BALANCE: Hybrid Autoregressive-Speculative LLM Inference at the Network Edge
  6. 2025Collaborative Speculative Inference for Efficient LLM Inference Serving
  7. 2026Lever: Speculative LLM Inference on Smartphones
  8. 2026AdaptiveSD A Stability-Aware, Runtime-Adaptive Speculative Decoding Framework with Multi-Policy Orchestration for CPU-Constrained LLM Inference