# MNN-LLM: A Generic Inference Engine for Fast Large Language Model Deployment on Mobile Devices

arXiv: https://arxiv.org/abs/2506.10443 · HTML: https://mlsanity.com/paper/2506.10443/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving.
- Builds on 8 of its 19 references that are in the map.
- First posted on arXiv 12 Jun 2025.

## Placed in

- Efficient LLM inference and serving / [Serving systems and request scheduling](https://mlsanity.com/inference/serving-systems-and-request-scheduling/)

## Builds on (most cited first, 8 of 8)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](https://mlsanity.com/paper/2210.17323/) (2022)
- [Root Mean Square Layer Normalization](https://mlsanity.com/paper/1910.07467/) (2019)
- [LLM in a flash: Efficient Large Language Model Inference with Limited Memory](https://mlsanity.com/paper/2312.11514/) (2023)
- [PowerInfer-2: Fast Large Language Model Inference on a Smartphone](https://mlsanity.com/paper/2406.06282/) (2024)
- [MNN: A Universal and Efficient Inference Engine](https://mlsanity.com/paper/2002.12418/) (2020)

## Built on by (most cited first, 8 of 8)

- [Context-Driven Performance Modeling for Causal Inference Operators on Neural Processing Units](https://mlsanity.com/paper/2509.25155/) (2025)
- [A Unified Model and Document Representation for On-Device Retrieval-Augmented Generation](https://mlsanity.com/paper/2604.14403/) (2026)
- [EdgeFM: Efficient Edge Inference for Vision-Language Models](https://mlsanity.com/paper/2604.27476/) (2026)
- [mzCache: On-Device LLM Memory Management under Multitasking](https://mlsanity.com/paper/2609.01338/) (2026)
- [EStream: Fast and Memory-Efficient MoE Prefill through Expert Virtualization on Mobile NPUs](https://mlsanity.com/paper/2609.06551/) (2026)
- [TierKV: Long-Context On-Device LLMs via Predictive Multi-Tier KV Caching](https://mlsanity.com/paper/2609.21172/) (2026)
- [Dynamic Flow, Static Graph: KV Cache Reuse for Efficient LLM Serving on Mobile NPUs](https://mlsanity.com/paper/2609.34727/) (2026)
- [SchemaWeaver: Accelerating On-Device Mobile Agentic AI via Schema-Guided Parallel LLM Decoding](https://mlsanity.com/paper/2610.17436/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have demonstrated exceptional performance across a variety of tasks. However, their substantial scale leads to significant computational resource consumption during inference, resulting in high costs. Consequently, edge device inference presents a promising solution. The primary challenges of edge inference include memory usage and inference speed. This paper introduces MNN-LLM, a framework specifically designed to accelerate the deployment of large language models on mobile devices. MNN-LLM addresses the runtime characteristics of LLMs through model quantization and DRAM-Flash hybrid storage, effectively reducing memory usage. It rearranges weights and inputs based on mobile CPU instruction sets and GPU characteristics while employing strategies such as multicore load balancing, mixed-precision floating-point operations, and geometric computations to enhance performance. Notably, MNN-LLM achieves up to a 8.6x speed increase compared to current mainstream LLM-specific frameworks.