ConsumerBench: Benchmarking Generative AI Applications on End-User Devices
Not on the map. Why?
Abstract (arXiv)
The recent shift in Generative AI (GenAI) applications from cloud-only environments to end-user devices introduces new challenges in resource management, system efficiency, and user experience. This paper presents ConsumerBench, a comprehensive benchmarking framework designed to evaluate the system efficiency and response time of GenAI models running on end-user devices. Unlike existing benchmarks that assume exclusive model access on dedicated GPUs, ConsumerBench simulates realistic multi-application scenarios executing concurrently on constrained hardware. Furthermore, ConsumerBench supports customizable workflows that simulate complex tasks requiring coordination among multiple applications. ConsumerBench captures both application-level metrics, including latency and Service Level Objective (SLO) attainment, and system-level metrics like CPU/GPU utilization and memory bandwidth. Through extensive experiments, ConsumerBench reveals inefficiencies in resource sharing, unfair scheduling under greedy allocation, and performance pitfalls of static model server configurations. The paper also provides practical insights for model developers and system designers, highlighting the benefits of custom kernels tailored to consumer-grade GPU architectures and the value of implementing SLO-aware scheduling strategies.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 22 of its 43 references that are in the map. First posted on arXiv 21 Jun 2025.
Builds on
22 of 43 references.
References · 22
- 2024Qwen2.5 Technical Reportcited by 4,942
- 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
- 2019Searching for MobileNetV3cited by 1,994
- 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
- 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
- 2018Attention U-Net: Learning Where to Look for the Pancreascited by 1,435
- 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
- 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 461
- 2023SnapFusion: Text-to-Image Diffusion Model on Mobile Devices within Two Secondscited by 227
- 2024StableToolBench: Towards Stable Large-Scale Benchmarking on Tool Learning of Large Language Modelscited by 132
- 2024AIOS: LLM Agent Operating Systemcited by 97
- 2021Earnings-21: A Practical Benchmark for ASR in the Wildcited by 62
10 more
- 2024Mobile-Bench: An Evaluation Benchmark for LLM-based Mobile Agentscited by 56
- 2024Andes: Defining and Enhancing Quality-of-Experience in LLM-Based Text Streaming Servicescited by 45
- 2024CRAB: Cross-environment Agent Benchmark for Multimodal Language Model Agentscited by 39
- 2024MELTing point: Mobile Evaluation of Language Transformerscited by 36
- 2023Turning Whisper into Real-Time Transcription Systemcited by 24
- 2024Uncertainty-Aware Hybrid Inference with On-Device Small and Remote Large Language Modelscited by 19
- 2023Specialist or Generalist? Instruction Tuning for Specific NLP Taskscited by 17
- 2024Benchmarking Edge AI Platforms for High-Performance ML Inferencecited by 9
- 2024PalmBench: A Comprehensive Benchmark of Compressed Large Language Models on Mobile Platformscited by 8
- 2023A Critical Review of Large Language Models: Sensitivity, Bias, and the Path Toward Specialized AIcited by 7
Built on by
Citing papers per year: 2025: 1; 2026: 1.
2 papers.