A Quantitative Review on Language Model Efficiency Research
Not on the map. Why?
Abstract (arXiv)
Language models (LMs) are being scaled and becoming powerful. Improving their efficiency is one of the core research topics in neural information processing systems. Tay et al. (2022) provided a comprehensive overview of efficient Transformers that have become an indispensable staple in the field of NLP. However, in the section of "On Evaluation", they left an open question "which fundamental efficient Transformer one should consider," answered by "still a mystery" because "many research papers select their own benchmarks." Unfortunately, there was not quantitative analysis about the performances of Transformers on any benchmarks. Moreover, state space models (SSMs) have demonstrated their abilities of modeling long-range sequences with non-attention mechanisms, which were not discussed in the prior review. This article makes a meta analysis on the results from a set of papers on efficient Transformers as well as those on SSMs. It provides a quantitative review on LM efficiency research and gives suggestions for future research.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 82 of its 96 references that are in the map. First posted on arXiv 28 May 2023.
Builds on
82 of 96 references.
References · 82
- 2017Attention Is All You Needcited by 59,932
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
- 2020Longformer: The Long-Document Transformercited by 3,362
- 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
- 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
- 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
- 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
- 2022Emergent Abilities of Large Language Modelscited by 2,483
- 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
38 more
- 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
- 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
- 2020Big Bird: Transformers for Longer Sequencescited by 1,673
- 2020Reformer: The Efficient Transformercited by 1,658
- 2020Rethinking Attention with Performerscited by 1,584
- 2019Generating Long Sequences with Sparse Transformerscited by 1,523
- 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
- 2018An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modelingcited by 1,447
- 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,281
- 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
- 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,100
- 2016HyperNetworkscited by 1,003
- 2020MPNet: Masked and Permuted Pre-training for Language Understandingcited by 973
- 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
- 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 832
- 2020Efficient Transformers: A Surveycited by 784
- 2017Reading Wikipedia to Answer Open-Domain Questionscited by 777
- 2022Simplified State Space Layers for Sequence Modelingcited by 776
- 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
- 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
- 2016Language Modeling with Gated Convolutional Networkscited by 688
- 2021Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layerscited by 607
- 2013Generating Sequences With Recurrent Neural Networkscited by 549
- 2018A Discourse-Aware Attention Model for Abstractive Summarization of Long Documentscited by 477
- 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 469
- 2021Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attentioncited by 468
- 2022On the Parameterization and Initialization of Diagonal State Space Modelscited by 418
- 2021Random Feature Attentioncited by 350
- 2022Diagonal State Spaces are as Effective as Structured State Spacescited by 350
- 2021FNet: Mixing Tokens with Fourier Transformscited by 343
- 2020Sparse Sinkhorn Attentioncited by 250
- 2017Challenges in Data-to-Document Generationcited by 226
- 2018Adaptive Input Representations for Neural Language Modelingcited by 226
- 2016A Decomposable Attention Model for Natural Language Inferencecited by 214
- 2019Pay Less Attention with Lightweight and Dynamic Convolutionscited by 211
- 2020Synthesizer: Rethinking Self-Attention in Transformer Modelscited by 204
- 2019Leveraging Pre-trained Checkpoints for Sequence Generation Taskscited by 182
- 2022Mega: Moving Average Equipped Gated Attentioncited by 178
50 most cited of 82 All 82 (JSON)
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2024A Survey of Transformer-based Language Models with Focus on Efficiency
- 2022Efficient Long Sequence Modeling via State Space Augmented Transformer
- 2025Speed Always Wins: A Survey on Efficient Architectures for Large Language Models
- 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Models
- 2025Pimba: A Processing-in-Memory Acceleration for Post-Transformer Large Language Model Serving
- 2023A Comprehensive Overview of Large Language Models
- 2024An Empirical Study of Mamba-based Language Models
- 2024Mamba-360: Survey of State Space Models as Transformer Alternative for Long Sequence Modelling: Methods, Applications, and Challenges