Data to 7 Oct 2026

MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices

arXiv 2004.02984 ↗First posted 6 Apr 2020cs.CL cs.LGcited by 476

Not on the map. Why?

Abstract (arXiv)

Natural Language Processing (NLP) has recently achieved great success by using huge pre-trained models with hundreds of millions of parameters. However, these models suffer from heavy model sizes and high latency such that they cannot be deployed to resource-limited mobile devices. In this paper, we propose MobileBERT for compressing and accelerating the popular BERT model. Like the original BERT, MobileBERT is task-agnostic, that is, it can be generically applied to various downstream NLP tasks via simple fine-tuning. Basically, MobileBERT is a thin version of BERT_LARGE, while equipped with bottleneck structures and a carefully designed balance between self-attentions and feed-forward networks. To train MobileBERT, we first train a specially designed teacher model, an inverted-bottleneck incorporated BERT_LARGE model. Then, we conduct knowledge transfer from this teacher to MobileBERT. Empirical studies show that MobileBERT is 4.3x smaller and 5.5x faster than BERT_BASE while achieving competitive results on well-known benchmarks. On the natural language inference tasks of GLUE, MobileBERT achieves a GLUEscore o 77.7 (0.6 lower than BERT_BASE), and 62 ms latency on a Pixel 4 phone. On the SQuAD v1.1/v2.0 question answering task, MobileBERT achieves a dev F1 score of 90.0/79.2 (1.5/2.1 higher than BERT_BASE).

Key facts. Cited by 476 papers in the map (data through 7 Oct 2026). First posted on arXiv 6 Apr 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 107; 2022: 99; 2023: 91; 2024: 65; 2025: 67; 2026: 47.

476 papers.

Sort:

By area:

2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 643
2022Efficiently Scaling Transformer Inferencecited by 529
2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 425
2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 411
2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 408
2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 400
2022Confident Adaptive Language Modelingcited by 297
2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 285
2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
2021I-BERT: Integer-only BERT Quantizationcited by 250
2021SPoT: Better Frozen Model Adaptation through Soft Prompt Transfercited by 235

38 more

2022ZeroGen: Efficient Zero-shot Learning via Dataset Generationcited by 209
2022Language model compression with weighted low-rank factorizationcited by 192
2021Block Pruning For Faster Transformerscited by 176
2022Structured Pruning Learns Compact and Accurate Modelscited by 167
2023Speculative Decoding with Big Little Decodercited by 163
2022A Fast Post-Training Pruning Framework for Transformerscited by 161
2021Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Bettercited by 153
2021MoEfication: Transformer Feed-forward Layers are Mixtures of Expertscited by 149
2022Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuningcited by 149
2023Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothingcited by 147
2021Understanding and Overcoming the Challenges of Efficient Transformer Quantizationcited by 139
2021Learned Token Pruning for Transformerscited by 138
2021SEED: Self-supervised Distillation For Visual Representationcited by 132
2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
2022The Optimal BERT Surgeon: Scalable and Accurate Second-Order Pruning for Large Language Modelscited by 115
2021Finetuning Pretrained Transformers into RNNscited by 94
2022Less is More: Task-aware Layer-wise Distillation for Language Model Compressioncited by 91
2023SkipDecode: Autoregressive Skip Decoding with Batching and Caching for Efficient LLM Inferencecited by 89
2022MiniViT: Compressing Vision Transformers with Weight Multiplexingcited by 88
2023Distil-Whisper: Robust Knowledge Distillation via Large-Scale Pseudo Labellingcited by 87
2021Sparse is Enough in Scaling Transformerscited by 86
2022Position: Considerations for Differentially Private Learning with Large-Scale Public Pretrainingcited by 86
2021Compressing Visual-linguistic Model via Knowledge Distillationcited by 85
2022Cramming: Training a Language Model on a Single GPU in One Daycited by 81
2023A Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4cited by 81
2022Compression of Generative Pre-trained Language Models via Quantizationcited by 79
2023Full Stack Optimization of Transformer Inference: a Surveycited by 79
2022Efficient Methods for Natural Language Processing: A Surveycited by 78
2021Revisiting Adversarial Robustness Distillation: Robust Soft Labels Make Student Bettercited by 77
2024A Complete Survey on LLM-based AI Chatbotscited by 71
2021Prune Once for All: Sparse Pre-Trained Language Modelscited by 66
2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 62
2021Annealing Knowledge Distillationcited by 61
2021Compute and Energy Consumption Trends in Deep Learning Inferencecited by 61
2024Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMscited by 61
2021Keyword Transformer: A Self-Attention Model for Keyword Spottingcited by 59
2021NAS-BERT: Task-Agnostic and Adaptive-Size BERT Compression with Neural Architecture Searchcited by 55
2024Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Applicationcited by 55

50 most cited of 476 All 476 (JSON)