mlsanityData to 7 Oct 2026

LeanStream: A Speculate-and-Refine Streaming Framework for Efficient on-Device LLM Inference

arXiv 2609.03079 ↗First posted 2 Sep 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

On-device LLM inference is attractive for privacy and responsiveness, but remains challenging on mobile and embedded devices because model weights far exceed available DRAM. Prior systems exploit activation sparsity and offload weights to SSD or flash storage, but face a fundamental systems trade-off: accurate sparse execution decisions require the latest context, whereas efficient computation-I/O overlap requires early prediction. As a result, existing designs either serialize execution or incur redundant weight fetches, extra computation, and large cache overheads. We present LeanStream, a streaming speculate-and-refine framework for efficient on-device LLM inference. LeanStream progressively refines computation, loading, and cache-retention priorities using partial GPU results, enabling fine-grained overlap between GPU execution and storage I/O. We implement LeanStream on both mobile and embedded platforms. Compared with prior on-device LLM inference systems, LeanStream reduces memory usage by 4.8× to 7.5× at the best throughput achieved by prior work, while further improving token generation throughput by 1.6× to 2.1×.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 32 of its 68 references that are in the map. First posted on arXiv 2 Sep 2026.

Builds on

32 of 68 references.

References · 32

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2025Qwen3 Technical Reportcited by 8,195
  3. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  4. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  5. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  6. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  7. 2018CoQA: A Conversational Question Answering Challengecited by 674
  8. 2021ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learningcited by 253
  9. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 247
  10. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Managementcited by 242
  11. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 210
  12. 2022SCROLLS: Standardized CompaRison Over Long Language Sequencescited by 170
20 more
  1. 2023LLM in a flash: Efficient Large Language Model Inference with Limited Memorycited by 170
  2. 2021MoEfication: Transformer Feed-forward Layers are Mixtures of Expertscited by 149
  3. 2016A Survey on Learning to Hashcited by 119
  4. 2015Practical and Optimal LSH for Angular Distancecited by 105
  5. 2024PowerInfer-2: Fast Large Language Model Inference on a Smartphonecited by 101
  6. 2024Training-Free Activation Sparsity in Large Language Modelscited by 51
  7. 2019SLIDE : In Defense of Smart Algorithms over Hardware Acceleration for Large-Scale Deep Learning Systemscited by 46
  8. 2015Optimal Data-Dependent Hashing for Approximate Near Neighborscited by 43
  9. 2024ProSparse: Introducing and Enhancing Intrinsic Activation Sparsity within Large Language Modelscited by 43
  10. 2021Multiplying Matrices Without Multiplyingcited by 30
  11. 2017DeepIoT: Compressing Deep Neural Network Structures for Sensing Systems with a Compressor-Critic Frameworkcited by 22
  12. 2016Scalable and Sustainable Deep Learning via Randomized Hashingcited by 17
  13. 2025D2MoE: Dual Routing and Dynamic Scheduling for Efficient On-Device MoE-based LLM Servingcited by 17
  14. 2022STI: Turbocharge NLP Inference at the Edge via Elastic Pipeliningcited by 14
  15. 2024Efficient LLM Inference using Dynamic Input Pruning and Cache-Aware Maskingcited by 10
  16. 2025TokenFlow: Responsive LLM Text Streaming Serving under Request Burst via Preemptive Schedulingcited by 10
  17. 2024LookupFFN: Making Transformers Compute-lite for CPU inferencecited by 8
  18. 2017Bolt: Accelerated Data Mining with Fast Vector Compressioncited by 7
  19. 2025DAF: An Efficient End-to-End Dynamic Activation Framework for on-Device DNN Trainingcited by 3
  20. 2026Physical Self-Supervised Learning: IMU Sensing without Manual Labelscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025MIST: A Co-Design Framework for Heterogeneous, Multi-Stage LLM Inference
  2. 2025LP-Spec: Leveraging LPDDR PIM for Efficient LLM Mobile Speculative Inference with Architecture-Dataflow Co-Optimization
  3. 2025LOIP:Collaborative Lossless LLM Inference Serving with Offloading-based Pipeline Parallelism on Edge Devices
  4. 2025Understanding Bottlenecks for Efficiently Serving LLM Inference With KV Offloading
  5. 2026SparKV: Overhead-Aware KV Cache Loading for Efficient On-Device LLM Inference
  6. 2026Lever: Speculative LLM Inference on Smartphones
  7. 2025SpecOffload: Unlocking Latent GPU Capacity for LLM Inference on Resource-Constrained Devices
  8. 2026mzCache: On-Device LLM Memory Management under Multitasking