# EdgeBERT: Sentence-Level Energy Optimizations for Latency-Aware Multi-Task NLP Inference

arXiv: https://arxiv.org/abs/2011.14203 · HTML: https://mlsanity.com/paper/2011.14203/

## Key facts

- Cited by 70 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Nov 2020.

## Built on by (most cited first, 30 of 70)

- [AdaViT: Adaptive Tokens for Efficient Vision Transformer](https://mlsanity.com/paper/2112.07658/) (2021)
- [QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving](https://mlsanity.com/paper/2405.04532/) (2024)
- [A Fast Post-Training Pruning Framework for Transformers](https://mlsanity.com/paper/2204.09656/) (2022)
- [DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiency](https://mlsanity.com/paper/2408.00741/) (2024)
- [Full Stack Optimization of Transformer Inference: a Survey](https://mlsanity.com/paper/2302.14017/) (2023)
- [Zeus: Understanding and Optimizing GPU Energy Consumption of DNN Training](https://mlsanity.com/paper/2208.06102/) (2022)
- [Efficient Methods for Natural Language Processing: A Survey](https://mlsanity.com/paper/2209.00099/) (2022)
- [A Survey on Green Deep Learning](https://mlsanity.com/paper/2111.05193/) (2021)
- [DFX: A Low-latency Multi-FPGA Appliance for Accelerating Transformer-based Text Generation](https://mlsanity.com/paper/2209.10797/) (2022)
- [Empowering Edge Intelligence: A Comprehensive Survey on On-Device AI Models](https://mlsanity.com/paper/2503.06027/) (2025)
- [LLMCad: Fast and Scalable On-device Large Language Model Inference](https://mlsanity.com/paper/2309.04255/) (2023)
- [Edge-Cloud Polarization and Collaboration: A Comprehensive Survey for AI](https://mlsanity.com/paper/2111.06061/) (2021)
- [Graph Transformers: A Survey](https://mlsanity.com/paper/2407.09777/) (2024)
- [Adaptable Butterfly Accelerator for Attention-based NNs via Hardware and Algorithm Co-design](https://mlsanity.com/paper/2209.09570/) (2022)
- [LLMServingSim: A HW/SW Co-Simulation Infrastructure for LLM Inference Serving at Scale](https://mlsanity.com/paper/2408.05499/) (2024)
- [An Algorithm-Hardware Co-Optimized Framework for Accelerating N:M Sparse Transformers](https://mlsanity.com/paper/2208.06118/) (2022)
- [Gradient-Free Structured Pruning with Unlabeled Data](https://mlsanity.com/paper/2303.04185/) (2023)
- [X-Former: In-Memory Acceleration of Transformers](https://mlsanity.com/paper/2303.07470/) (2023)
- [Predictive Exit: Prediction of Fine-Grained Early Exits for Computation- and Energy-Efficient Inference](https://mlsanity.com/paper/2206.04685/) (2022)
- [EdgeFormer: A Parameter-Efficient Transformer for On-Device Seq2seq Generation](https://mlsanity.com/paper/2202.07959/) (2022)
- [AdapLeR: Speeding up Inference by Adaptive Length Reduction](https://mlsanity.com/paper/2203.08991/) (2022)
- [STI: Turbocharge NLP Inference at the Edge via Elastic Pipelining](https://mlsanity.com/paper/2207.05022/) (2022)
- [HADAS: Hardware-Aware Dynamic Neural Architecture Search for Edge Performance Scaling](https://mlsanity.com/paper/2212.03354/) (2022)
- [A Hardware Evaluation Framework for Large Language Model Inference](https://mlsanity.com/paper/2312.03134/) (2023)
- [Intelligence per Watt: Measuring Intelligence Efficiency of Local AI](https://mlsanity.com/paper/2511.07885/) (2025)
- [Pipeline Parallelism for Inference on Heterogeneous Edge Computing](https://mlsanity.com/paper/2110.14895/) (2021)
- [Differentially Private Model Compression](https://mlsanity.com/paper/2206.01838/) (2022)
- [CHAI: Clustered Head Attention for Efficient LLM Inference](https://mlsanity.com/paper/2403.08058/) (2024)
- [Sparse Attention Acceleration with Synergistic In-Memory Pruning and On-Chip Recomputation](https://mlsanity.com/paper/2209.00606/) (2022)
- [MX+: Pushing the Limits of Microscaling Formats for Efficient Large Language Model Serving](https://mlsanity.com/paper/2510.14557/) (2025)

## Abstract (arXiv)

Transformer-based language models such as BERT provide significant accuracy improvement for a multitude of natural language processing (NLP) tasks. However, their hefty computational and memory demands make them challenging to deploy to resource-constrained edge platforms with strict latency requirements. We present EdgeBERT, an in-depth algorithm-hardware co-design for latency-aware energy optimization for multi-task NLP. EdgeBERT employs entropy-based early exit predication in order to perform dynamic voltage-frequency scaling (DVFS), at a sentence granularity, for minimal energy consumption while adhering to a prescribed target latency. Computation and memory footprint overheads are further alleviated by employing a calibrated combination of adaptive attention span, selective network pruning, and floating-point quantization. Furthermore, in order to maximize the synergistic benefits of these algorithms in always-on and intermediate edge computing settings, we specialize a 12nm scalable hardware accelerator system, integrating a fast-switching low-dropout voltage regulator (LDO), an all-digital phase-locked loop (ADPLL), as well as, high-density embedded non-volatile memories (eNVMs) wherein the sparse floating-point bit encodings of the shared multi-task parameters are carefully stored. Altogether, latency-aware multi-task NLP inference acceleration on the EdgeBERT hardware system generates up to 7x, 2.5x, and 53x lower energy compared to the conventional inference without early stopping, the latency-unbounded early exit approach, and CUDA adaptations on an Nvidia Jetson Tegra X2 mobile GPU, respectively.