# On-Device Neural Net Inference with Mobile GPUs

arXiv: https://arxiv.org/abs/1907.01989 · HTML: https://mlsanity.com/paper/1907.01989/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Jul 2019.

## Built on by (most cited first, 30 of 32)

- [Device-Cloud Collaborative Learning for Recommendation](https://mlsanity.com/paper/2104.06624/) (2021)
- [POET: Training Neural Networks on Tiny Devices with Integrated Rematerialization and Paging](https://mlsanity.com/paper/2207.07697/) (2022)
- [Smart at what cost? Characterising Mobile Deep Neural Networks in the wild](https://mlsanity.com/paper/2109.13963/) (2021)
- [Green Edge AI: A Contemporary Survey](https://mlsanity.com/paper/2312.00333/) (2023)
- [Fast and Accurate Single-Image Depth Estimation on Mobile Devices, Mobile AI 2021 Challenge: Report](https://mlsanity.com/paper/2105.08630/) (2021)
- [Real-Time Video Super-Resolution on Smartphones with Deep Learning, Mobile AI 2021 Challenge: Report](https://mlsanity.com/paper/2105.08826/) (2021)
- [PatrickStar: Parallel Training of Pre-trained Models via Chunk-based Memory Management](https://mlsanity.com/paper/2108.05818/) (2021)
- [MFAGAN: A Compression Framework for Memory-Efficient On-Device Super-Resolution GAN](https://mlsanity.com/paper/2107.12679/) (2021)
- [Layer Pruning on Demand with Intermediate CTC](https://mlsanity.com/paper/2106.09216/) (2021)
- [The Effect of Model Compression on Fairness in Facial Expression Recognition](https://mlsanity.com/paper/2201.01709/) (2022)
- [Learned Smartphone ISP on Mobile GPUs with Deep Learning, Mobile AI & AIM 2022 Challenge: Report](https://mlsanity.com/paper/2211.03885/) (2022)
- [MicroISP: Processing 32MP Photos on Mobile Devices with Deep Learning](https://mlsanity.com/paper/2211.06770/) (2022)
- [FastICARL: Fast Incremental Classifier and Representation Learning with Efficient Budget Allocation in Audio Sensing Applications](https://mlsanity.com/paper/2106.07268/) (2021)
- [BasisNet: Two-stage Model Synthesis for Efficient Inference](https://mlsanity.com/paper/2105.03014/) (2021)
- [Federated Remote Physiological Measurement with Imperfect Data](https://mlsanity.com/paper/2203.05759/) (2022)
- [PyNet-V2 Mobile: Efficient On-Device Photo Processing With Neural Networks](https://mlsanity.com/paper/2211.06263/) (2022)
- [ML-EXray: Visibility into ML Deployment on the Edge](https://mlsanity.com/paper/2111.04779/) (2021)
- [SYENet: A Simple Yet Effective Network for Multiple Low-Level Vision Tasks with Real-time Performance on Mobile Device](https://mlsanity.com/paper/2308.08137/) (2023)
- [PolyThrottle: Energy-efficient Neural Network Inference on Edge Devices](https://mlsanity.com/paper/2310.19991/) (2023)
- [Memory Planning for Deep Neural Networks](https://mlsanity.com/paper/2203.00448/) (2022)
- [Scaling On-Device GPU Inference for Large Generative Models](https://mlsanity.com/paper/2505.00232/) (2025)
- [Efficient Heterogeneous Video Segmentation at the Edge](https://mlsanity.com/paper/2208.11666/) (2022)
- [Realistic Bokeh Effect Rendering on Mobile GPUs, Mobile AI & AIM 2022 challenge: Report](https://mlsanity.com/paper/2211.06769/) (2022)
- [A Framework for Event-based Computer Vision on a Mobile Device](https://mlsanity.com/paper/2205.06836/) (2022)
- [An efficient and flexible inference system for serving heterogeneous ensembles of deep neural networks](https://mlsanity.com/paper/2208.14049/) (2022)
- [Inference Latency Prediction at the Edge](https://mlsanity.com/paper/2210.02620/) (2022)
- [NeuroFlux: Memory-Efficient CNN Training Using Adaptive Local Learning](https://mlsanity.com/paper/2402.14139/) (2024)
- [t-READi: Transformer-Powered Robust and Efficient Multimodal Inference for Autonomous Driving](https://mlsanity.com/paper/2410.09747/) (2024)
- [Accelerating Mobile Inference through Fine-Grained CPU-GPU Co-Execution](https://mlsanity.com/paper/2510.21081/) (2025)
- [CompactFlowNet: Efficient Real-time Optical Flow Estimation on Mobile Devices](https://mlsanity.com/paper/2412.13273/) (2024)

## Abstract (arXiv)

On-device inference of machine learning models for mobile phones is desirable due to its lower latency and increased privacy. Running such a compute-intensive task solely on the mobile CPU, however, can be difficult due to limited computing power, thermal constraints, and energy consumption. App developers and researchers have begun exploiting hardware accelerators to overcome these challenges. Recently, device manufacturers are adding neural processing units into high-end phones for on-device inference, but these account for only a small fraction of hand-held devices. In this paper, we present how we leverage the mobile GPU, a ubiquitous hardware accelerator on virtually every phone, to run inference of deep neural networks in real-time for both Android and iOS devices. By describing our architecture, we also discuss how to design networks that are mobile GPU-friendly. Our state-of-the-art mobile GPU inference engine is integrated into the open-source project TensorFlow Lite and publicly available at https://tensorflow.org/lite.