mlsanityData to 7 Oct 2026

Lever: Speculative LLM Inference on Smartphones

arXiv 2605.16786 ↗First posted 16 May 2026cs.LGcited by 1

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

Large language models (LLMs) are increasingly needed for interactive mobile applications, but high-quality models exceed the limited DRAM available on smartphones. Flash storage can hold larger models, yet flash-backed inference is slow because autoregressive decoding repeatedly invokes the target model and incurs costly I/O. We observe that speculative decoding is a natural fit for this setting: a small draft model can remain in DRAM, while a larger flash-resident target model verifies multiple candidate tokens per invocation. However, existing methods assume server-class accelerators and fail to account for prolonged I/O latency, limited computation parallelism, and irregular speculation execution. We present Lever, an end-to-end system for efficient flash-backed LLM inference on smartphones. Lever jointly optimizes the three stages of speculative decoding under mobile constraints. For drafting, it builds token trees using an I/O- and compute-aware gain-cost objective. For verification, it prunes low-value branches through early-exit prediction to reduce target-model computation. For execution, it maps speculation efficiently across mobile CPU-NPU hardware to improve utilization. Comprehensive evaluations show that Lever reduces inference latency by an average of 2.93x over baseline flash-offloaded inference and 1.50x over conventional speculative decoding, narrowing the latency gap between flash-backed and memory-resident LLM inference.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 30 of its 39 references that are in the map. First posted on arXiv 16 May 2026.

Builds on

30 of 39 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test3 Mar
  2. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees24 Jun
  3. 2024Hydra: Sequentially-Dependent Draft Heads for Medusa Decoding7 Feb
  4. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decoding3 Feb
  5. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty26 Jan
  6. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads19 Jan
  7. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification16 May
  8. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
  9. 2022Fast Inference from Transformers via Speculative Decoding30 Nov

Other references · 21

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2021Program Synthesis with Large Language Modelscited by 3,509
  3. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  4. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  5. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 643
  6. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  7. 2020DeeBERT: Dynamic Early Exiting for Accelerating BERT Inferencecited by 322
  8. 2018Blockwise Parallel Decoding for Deep Autoregressive Modelscited by 318
  9. 2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decodingcited by 302
  10. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 247
  11. 2020FastBERT: a Self-distilling BERT with Adaptive Inference Timecited by 224
  12. 2024MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Casescited by 216
9 more
  1. 2023BitNet: Scaling 1-bit Transformers for Large Language Modelscited by 175
  2. 2023LLM in a flash: Efficient Large Language Model Inference with Limited Memorycited by 170
  3. 2024PowerInfer-2: Fast Large Language Model Inference on a Smartphonecited by 101
  4. 2024CE-CoLLM: Efficient and Adaptive Large Language Models Through Cloud-Edge Collaborationcited by 25
  5. 2025PRISM: Privacy-Aware Routing for Adaptive Cloud-Edge LLM Inference via Semantic Sketch Collaborationcited by 10
  6. 2024A First Look At Efficient And Secure On-Device LLM Inference Against KV Leakagecited by 8
  7. 2024Neuralink: Fast LLM Inference on Smartphones with Neuron Co-Activation Linkingcited by 4
  8. 2025DynaKV: Enabling Accurate and Efficient Long-Sequence LLM Decoding on Smartphonescited by 4
  9. 2025Long Exposure: Accelerating Parameter-Efficient Fine-Tuning for LLMs under Shadowy Sparsitycited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026AS2D: Accelerating On-Demand Audio Understanding on Mobile Devicescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Accelerating LLM Inference with Staged Speculative Decoding
  2. 2025SLED: A Speculative LLM Decoding Framework for Efficient Edge Serving
  3. 2025MNN-LLM: A Generic Inference Engine for Fast Large Language Model Deployment on Mobile Devices
  4. 2024SpecExec: Massively Parallel Speculative Decoding for Interactive LLM Inference on Consumer Devices
  5. 2025Mirror Speculative Decoding: Breaking the Serial Barrier in LLM Inference
  6. 2026BigMoMo: Efficient Inference of Large-Scale MoE with Speculative Decoding on Mobile Devices
  7. 2026BALANCE: Hybrid Autoregressive-Speculative LLM Inference at the Network Edge
  8. 2026BitNest: Bit-Nested Speculative Decoding for Memory-Efficient LLM Inference Acceleration