# Real-Time Execution of Large-scale Language Models on Mobile

arXiv: https://arxiv.org/abs/2009.06823 · HTML: https://mlsanity.com/paper/2009.06823/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Sep 2020.

## Built on by (most cited first, 2 of 2)

- [Automatic Mapping of the Best-Suited DNN Pruning Schemes for Real-Time Mobile Acceleration](https://mlsanity.com/paper/2111.11581/) (2021)
- [Towards Fast and Accurate Multi-Person Pose Estimation on Mobile Devices](https://mlsanity.com/paper/2106.15304/) (2021)

## Abstract (arXiv)

Pre-trained large-scale language models have increasingly demonstrated high accuracy on many natural language processing (NLP) tasks. However, the limited weight storage and computational speed on hardware platforms have impeded the popularity of pre-trained models, especially in the era of edge computing. In this paper, we seek to find the best model structure of BERT for a given computation size to match specific devices. We propose the first compiler-aware neural architecture optimization framework. Our framework can guarantee the identified model to meet both resource and real-time specifications of mobile devices, thus achieving real-time execution of large transformer-based models like BERT variants. We evaluate our model on several NLP tasks, achieving competitive results on well-known benchmarks with lower latency on mobile devices. Specifically, our model is 5.2x faster on CPU and 4.1x faster on GPU with 0.5-2% accuracy loss compared with BERT-base. Our overall framework achieves up to 7.8x speedup compared with TensorFlow-Lite with only minor accuracy loss.