Elastic On-Device LLM Service
Not on the map. Why?
Abstract (arXiv)
On-device Large Language Models (LLMs) are transforming mobile AI, catalyzing applications like UI automation without privacy concerns. Nowadays the common practice is to deploy a single yet powerful LLM as a general task solver for multiple requests. We identify a key system challenge in this paradigm: current LLMs lack the elasticity to serve requests that have diversified Service-Level Objectives (SLOs) on inference latency. To tackle this, we present sys, an on-device LLM service that elasticizes both the model and the prompt dimension of a full LLM. It incorporates (1) a one-shot neuron-reordering method, which leverages the intrinsic permutation consistency in transformer models to generate high-quality elasticized sub-models with minimal runtime switching overhead; (2) a dual-head tiny language model, which efficiently and effectively refines the prompt and orchestrates the elastification between model and prompt. We implement such an elastic on-device LLM service on multiple COTS smartphones, and evaluate sys on both standalone NLP/mobile-agent datasets and end-to-end synthesized traces. On diverse SLOs, sys outperforms 7 strong baselines in (absolute) accuracy by up to 14.83% and 10.45% on average, with <1% TTFT switching overhead, on-par memory consumption and <100 offline GPU hours.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 56 of its 81 references that are in the map. First posted on arXiv 8 Sep 2024.
Builds on
56 of 81 references.
References · 56
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
- 2024The Llama 3 Herd of Modelscited by 14,192
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
- 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
- 2020Measuring Massive Multitask Language Understandingcited by 7,768
- 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
- 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
- 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
- 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
38 more
- 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
- 2017Axiomatic Attribution for Deep Networkscited by 3,350
- 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
- 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
- 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
- 2023MemGPT: Towards LLMs as Operating Systemscited by 1,295
- 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
- 2016Pruning Filters for Efficient ConvNetscited by 1,129
- 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
- 2019Once-for-All: Train One Network and Specialize it for Efficient Deploymentcited by 778
- 2017Crowdsourcing Multiple Choice Science Questionscited by 742
- 2022A Survey on In-context Learningcited by 717
- 2017BranchyNet: Fast Inference via Early Exiting from Deep Neural Networkscited by 591
- 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 476
- 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compressioncited by 443
- 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 408
- 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 369
- 2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 369
- 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verificationcited by 359
- 2024LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compressioncited by 343
- 2023LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Modelscited by 279
- 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 257
- 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 247
- 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 210
- 2023AutoDroid: LLM-powered Task Automation in Androidcited by 178
- 2023The Learnability of In-Context Learningcited by 126
- 2023LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructionscited by 124
- 2024LaCo: Large Language Model Pruning via Layer Collapsecited by 124
- 2024PowerInfer-2: Fast Large Language Model Inference on a Smartphonecited by 101
- 2024What Matters in Transformers? Not All Attention is Neededcited by 83
- 2020EdgeBERT: Sentence-Level Energy Optimizations for Latency-Aware Multi-Task NLP Inferencecited by 69
- 2018NestDNN: Resource-Aware Multi-Tenant On-Device Deep Learning for Continuous Mobile Visioncited by 68
- 2020MNN: A Universal and Efficient Inference Enginecited by 64
- 2023LLMCad: Fast and Scalable On-device Large Language Model Inferencecited by 45
- 2024Octopus v2: On-device language model for super agentcited by 45
- 2024LlamaTouch: A Faithful and Scalable Testbed for Mobile UI Task Automationcited by 36
- 2023LLM as OS, Agents as Apps: Envisioning AIOS, Agents and the AIOS-Agent Ecosystemcited by 35
- 2023PIT: Optimization of Dynamic Sparse Deep Learning Models via Permutation Invariant Transformationcited by 25
50 most cited of 56 All 56 (JSON)
Built on by
Citing papers per year: 2024: 1; 2025: 1; 2026: 1.
3 papers.