mlsanityData to 14 Oct 2026

mzCache: On-Device LLM Memory Management under Multitasking

arXiv 2609.01338 ↗First posted 1 Sep 2026cs.OS cs.DC cs.LGcited by 0

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

On-device mobile Large Language Model (LLM) inference is gaining significant attention. However, mobile devices operate in highly dynamic multitasking environments where users frequently switch between applications. This creates memory pressure, forcing LLM memory (model weights and KV cache) to be evicted by the operating system. When a new inference request arrives, the inference system must restore the evicted memory through slow storage reads or recompute the entire KV cache, severely degrading responsiveness. To address this, we present mzCache, an on-device LLM inference system with specialized memory management for multitasking environments. Under unpredictable memory pressure, mzCache elastically evicts LLM memory and leverages the unified memory of mobile SoCs to enable zero-wait inference on the GPU with concurrent CPU-side restoration. mzCache realizes this through restoration-oriented memory management: LLM memory is partitioned into fine-grained shared buffers to enable partial eviction and restoration with concurrent cross-processor access, while hybrid swap and backward-out eviction policies ensure low-latency restoration from any eviction state. Implemented on llama.cpp and deployed as an Android application, mzCache achieves 2.1-5.5× reduction in Time-to-First-Token compared to storage-backed partial offload and demonstrates its effectiveness in real multitasking scenarios.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 26 of its 58 references that are in the map. First posted on arXiv 1 Sep 2026.

Builds on

26 of 58 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025MNN-LLM: A Generic Inference Engine for Fast Large Language Model Deployment on Mobile Devices12 Jun

Other references · 25

  1. 2017Attention Is All You Needcited by 60,361
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  4. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  5. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  6. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  7. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  8. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  9. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  10. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 501
  11. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Managementcited by 245
  12. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 217
13 more
  1. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Servingcited by 203
  2. 2024PowerInfer-2: Fast Large Language Model Inference on a Smartphonecited by 103
  3. 2024A Review on Edge Large Language Models: Design, Execution, and Applicationscited by 100
  4. 2024KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantizationcited by 76
  5. 2023EdgeMoE: Empowering Sparse Large Language Models on Mobile Devicescited by 69
  6. 2024Fast On-device LLM Inference with NPUscited by 62
  7. 2023Stateful Large Language Model Serving with Pensievecited by 61
  8. 2024Fast State Restoration in LLM Serving with HCachecited by 33
  9. 2025EXAONE 4.0: Unified Large Language Models Integrating Non-reasoning and Reasoning Modescited by 32
  10. 2025Characterizing Mobile SoC for Accelerating Heterogeneous LLM Inferencecited by 22
  11. 2024LLM for Mobile: An Initial Roadmapcited by 11
  12. 2025Scaling LLM Test-Time Compute with Mobile NPU on Smartphonescited by 11
  13. 2025Ariadne: A Hotness-Aware and Size-Adaptive Compressed Swap Technique for Fast Application Relaunch and Reduced CPU Usage on Mobile Devicescited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026TierKV: Long-Context On-Device LLMs via Predictive Multi-Tier KV Caching
  2. 2026KVDrive: A Holistic Multi-Tier KV Cache Management System for Long-Context LLM Inference
  3. 2025LLaMCAT: Optimizing Large Language Model Inference with Cache Arbitration and Throttling
  4. 2026LeanStream: A Speculate-and-Refine Streaming Framework for Efficient on-Device LLM Inference
  5. 2025Accelerating LLM Inference via Dynamic KV Cache Placement in Heterogeneous Memory System
  6. 2024Harnessing Your DRAM and SSD for Sustainable and Accessible LLM Inference with Mixed-Precision and Multi-level Caching
  7. 2025Architecting Long-Context LLM Acceleration with Packing-Prefetch Scheduler and Ultra-Large Capacity On-Chip Memories
  8. 2026Lever: Speculative LLM Inference on Smartphones