Data to 14 Oct 2026

Towards Efficient NLP: A Standard Evaluation and A Strong Baseline

arXiv 2110.07038 ↗First posted 13 Oct 2021cs.CL cs.AIcited by 44

Not on the map. Why?

Abstract (arXiv)

Supersized pre-trained language models have pushed the accuracy of various natural language processing (NLP) tasks to a new state-of-the-art (SOTA). Rather than pursuing the reachless SOTA accuracy, more and more researchers start paying attention on model efficiency and usability. Different from accuracy, the metric for efficiency varies across different studies, making them hard to be fairly compared. To that end, this work presents ELUE (Efficient Language Understanding Evaluation), a standard evaluation, and a public leaderboard for efficient NLP models. ELUE is dedicated to depict the Pareto Frontier for various language understanding tasks, such that it can tell whether and how much a method achieves Pareto improvement. Along with the benchmark, we also release a strong baseline, ElasticBERT, which allows BERT to exit at any layer in both static and dynamic ways. We demonstrate the ElasticBERT, despite its simplicity, outperforms or performs on par with SOTA compressed and early exiting models. With ElasticBERT, the proposed ELUE has a strong Pareto Frontier and makes a better evaluation for efficient NLP models.

Key facts. Cited by 44 papers in the map (data through 14 Oct 2026). Builds on 52 of its 66 references that are in the map. First posted on arXiv 13 Oct 2021.

Builds on

52 of 66 references.

References · 52

2017Attention Is All You Needcited by 60,361
2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
2014Adam: A Method for Stochastic Optimizationcited by 37,240
2020Language Models are Few-Shot Learnerscited by 32,419
2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811

38 more

2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
2015A large annotated corpus for learning natural language inferencecited by 1,847
2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,043
2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 1,001
2019Are Sixteen Heads Really Better than One?cited by 816
2020Long Range Arena: A Benchmark for Efficient Transformerscited by 700
2018Universal Transformerscited by 647
2019Patient Knowledge Distillation for BERT Model Compressioncited by 508
2016Adaptive Computation Time for Recurrent Neural Networkscited by 506
2020Pre-trained Models for Natural Language Processing: A Surveycited by 505
2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 482
2021Dynabench: Rethinking Benchmarking in NLPcited by 459
2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 439
2021A Survey of Transformerscited by 401
2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 359
2020DeeBERT: Dynamic Early Exiting for Accelerating BERT Inferencecited by 332
2018SentEval: An Evaluation Toolkit for Universal Sentence Representationscited by 294
2020CLUE: A Chinese Language Understanding Evaluation Benchmarkcited by 278
2020BERT Loses Patience: Fast and Robust Inference with Early Exitcited by 273
2021The GEM Benchmark: Natural Language Generation, its Evaluation and Metricscited by 239
2018The Natural Language Decathlon: Multitask Learning as Question Answeringcited by 236
2019Depth-Adaptive Transformercited by 236
2020FastBERT: a Self-distilling BERT with Adaptive Inference Timecited by 226
2020Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learningcited by 185
2020The Right Tool for the Job: Matching Model and Instance Complexitiescited by 126
2019Well-Read Students Learn Better: On the Importance of Pre-training Compact Modelscited by 120
2020BERT-of-Theseus: Compressing BERT by Progressive Module Replacingcited by 113
2020CoLAKE: Contextualized Language and Knowledge Embeddingcited by 86
2019Improved Techniques for Training Adaptive Deep Networkscited by 66
2021NeurIPS 2020 EfficientQA Competition: Systems, Analyses and Lessons Learnedcited by 52
2020CascadeBERT: Accelerating Inference of Pre-trained Language Models via Calibrated Complete Models Cascadecited by 47
2013Low-Rank Approximations for Conditional Feedforward Computation in Deep Neural Networkscited by 44
2021Paradigm Shift in Natural Language Processingcited by 39

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 14; 2023: 8; 2024: 11; 2025: 8; 2026: 3.

44 papers.

Sort:

By area:

2022Confident Adaptive Language Modelingcited by 310
2023A Survey on Model Compression for Large Language Modelscited by 269
2024A Comprehensive Study of Knowledge Editing for Large Language Modelscited by 163
2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 152
2022Efficient Methods for Natural Language Processing: A Surveycited by 78
2022A Survey on Model Compression and Acceleration for Pretrained Language Modelscited by 45
2022A Simple Hash-Based Early Exiting Approach For Language Understanding and Generationcited by 32
2022Square One Bias in NLP: Towards a Multi-Dimensional Exploration of the Research Manifoldcited by 29
2023The Benefits of Bad Advice: Autocontrastive Decoding across Model Layerscited by 29
2022A Survey on Dynamic Neural Networks for Natural Language Processingcited by 24
2022Late Prompt Tuning: A Late Prompt Could Be Better Than Many Promptscited by 23
2022pNLP-Mixer: an Efficient all-MLP Architecture for Languagecited by 19

32 more

2023SplitEE: Early Exit in Deep Neural Networks with Split Computingcited by 13
2024CEEBERT: Cross-Domain Inference in Early Exit BERTcited by 12
2025FREE: Fast and Robust Vision Language Models with Early Exitscited by 12
2022VLUE: A Multi-Task Benchmark for Evaluating Vision-Language Modelscited by 11
2022Unsupervised Early Exit in DNNs with Multiple Exitscited by 10
2024I-SplitEE: Image classification in Split Computing DNNs with Early Exitscited by 9
2024MOPO: Multi-Objective Prompt Optimization for Affective Text Generationcited by 9
2023Efficiency Pentathlon: A Standardized Arena for Efficiency Evaluationcited by 7
2025BEEM: Boosting Performance of Early Exit DNNs using Multi-Exit Classifiers as Expertscited by 6
2022MiniDisc: Minimal Distillation Schedule for Language Model Compressioncited by 5
2023Finding the SWEET Spot: Analysis and Improvement of Adaptive Inference in Low Resource Settingscited by 5
2024DAdEE: Unsupervised Domain Adaptation in Early Exit PLMscited by 5
2022Moving Beyond Downstream Task Accuracy for Information Retrieval Benchmarkingcited by 4
2024RAEE: A Robust Retrieval-Augmented Early Exit Framework for Efficient Inferencecited by 4
2024Expanding Expressivity in Transformer Models with M"obiusAttentioncited by 4
2024Distributed Inference on Mobile Edge and Cloud: An Early Exit based Clustering Approachcited by 3
2025Beyond Greedy Exits: Improved Early Exit Decisions for Risk Control and Reliabilitycited by 3
2025Generalizable and Efficient Automated Scoring with a Knowledge-Distilled Multi-Task Mixture-of-Expertscited by 3
2022COST-EFF: Collaborative Optimization of Spatial and Temporal Efficiency with Slenderized Multi-exit Language Modelscited by 2
2024Multimodal Adaptive Inference for Document Image Classification with Anytime Early Exitingcited by 2
2025InversionGNN: A Dual Path Network for Multi-Property Molecular Optimizationcited by 2
2026EdgeNav-QE: QLoRA Quantization and Dynamic Early Exit for LAM-based Navigation on Edge Devicescited by 2
2022Challenges in Explanation Quality Evaluationcited by 1
2025Comparing energy consumption and accuracy in text classification inferencecited by 1
2026AgentCompress: Task-Aware Compression for Affordable Large Language Model Agentscited by 1
2023Test-Time Adaptation with Perturbation Consistency Learningcited by 0
2023Generate to Understand for Representationcited by 0
2023On Elastic Language Modelscited by 0
2024BEExformer: A Fast Inferencing Binarized Transformer with Early Exitscited by 0
2025Know What You Don't Know: Selective Prediction for Early Exit DNNscited by 0
2025Balancing Knowledge Updates: Toward Unified Modular Editing in LLMscited by 0
2026FLOPs vs Real Work: The Importance of Replication in AI Efficiency Assessmentcited by 0

Similar, not cited

Close in wording, no citation either way