Processing Natural Language on Embedded Devices: How Well Do Transformer Models Perform?
Not on the map. Why?
Abstract (arXiv)
This paper presents a performance study of transformer language models under different hardware configurations and accuracy requirements and derives empirical observations about these resource/accuracy trade-offs. In particular, we study how the most commonly used BERT-based language models (viz., BERT, RoBERTa, DistilBERT, and TinyBERT) perform on embedded systems. We tested them on four off-the-shelf embedded platforms (Raspberry Pi, Jetson, UP2, and UDOO) with 2 GB and 4 GB memory (i.e., a total of eight hardware configurations) and four datasets (i.e., HuRIC, GoEmotion, CoNLL, WNUT17) running various NLP tasks. Our study finds that executing complex NLP tasks (such as "sentiment" classification) on embedded systems is feasible even without any GPUs (e.g., Raspberry Pi with 2 GB of RAM). Our findings can help designers understand the deployability and performance of transformer language models, especially those based on BERT architectures.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 19 of its 46 references that are in the map. First posted on arXiv 23 Apr 2023.
Builds on
19 of 46 references.
References · 19
- 2017Attention Is All You Needcited by 59,932
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
- 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
- 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
- 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
- 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
- 2019Are Sixteen Heads Really Better than One?cited by 803
- 2019Patient Knowledge Distillation for BERT Model Compressioncited by 502
- 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 476
- 2019Q8BERT: Quantized 8Bit BERTcited by 296
7 more
- 2019Distilling Task-Specific Knowledge from BERT into Simple Neural Networkscited by 152
- 2019Well-Read Students Learn Better: On the Importance of Pre-training Compact Modelscited by 118
- 2018From Eliza to XiaoIce: Challenges and Opportunities with Social Chatbotscited by 85
- 2020EdgeBERT: Sentence-Level Energy Optimizations for Latency-Aware Multi-Task NLP Inferencecited by 69
- 2020SqueezeBERT: What can computer vision teach NLP about efficient neural networks?cited by 50
- 2019Small and Practical BERT Models for Sequence Labelingcited by 40
- 2020The Microsoft Toolkit of Multi-Task Deep Neural Networks for Natural Language Understandingcited by 28
Built on by
Citing papers per year: 2025: 2.
2 papers.