mlsanityData to 14 Oct 2026

A Survey of Transformer-based Language Models with Focus on Efficiency

arXiv 2406.16893 ↗First posted 15 May 2024cs.CL cs.AIcited by 6

Not on the map. Why?

Abstract (arXiv)

The emergence of Transformer-based Large Language Models (LLMs) has substantially augmented the capabilities of Natural Language Processing (NLP), thereby intensifying the demand for computational resources. Therefore, enhancing efficiency based on factors like computational requirements, energy consumption, carbon footprint and financial cost has become a vital area of research. This motivates us to conduct a survey on Transformer-based LLMs in NLP from the perspective of efficiency. In this survey of 312 articles, the efficiency-improvement endeavors have been systematically discussed targeting various aspects such as data curation, model design, model downsizing, and dynamic inferencing. This has been augmented with efficiency considerations in model adaptation strategies like pre-training, fine-tuning, prompt-engineering and Retrieval-Augmented Generation (RAG). Furthermore, a statistical analysis followed by an in-depth evaluation of the efficiency and efficacy of more than 30 renowned NLP models has been performed on 13 evaluation benchmarks. This paper offers valuable insights for researchers, professionals as well as scholars, and explores the trend of research toward sustainable practices in NLP.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 51 of its 321 references that are in the map. First posted on arXiv 15 May 2024.

Builds on

51 of 321 references.

References · 51

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2020Scaling Laws for Neural Language Modelscited by 6,586
  6. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  7. 2020Longformer: The Long-Document Transformercited by 3,413
  8. 2023Mistral 7Bcited by 3,164
  9. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  10. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  11. 2024Mixtral of Expertscited by 1,823
  12. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
38 more
  1. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  2. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  3. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  4. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  5. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 1,006
  6. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 812
  7. 2022BERTopic: Neural topic modeling with a class-based TF-IDF procedurecited by 801
  8. 2022Galactica: A Large Language Model for Sciencecited by 744
  9. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
  10. 2014Neural Turing Machinescited by 644
  11. 2021Carbon Emissions and Large Neural Network Trainingcited by 567
  12. 2021ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generationcited by 260
  13. 2018A Simple Method for Commonsense Reasoningcited by 223
  14. 2024A Survey on Efficient Inference for Large Language Modelscited by 205
  15. 2024Corrective Retrieval Augmented Generationcited by 197
  16. 2023BitNet: Scaling 1-bit Transformers for Large Language Modelscited by 189
  17. 2023ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Modelscited by 169
  18. 2023Walking Down the Memory Maze: Beyond Context Limit through Interactive Readingcited by 133
  19. 2023RPTQ: Reorder-based Post-training Quantization for Large Language Modelscited by 116
  20. 2024How to Train Data-Efficient LLMscited by 93
  21. 2023SkipDecode: Autoregressive Skip Decoding with Batching and Caching for Efficient LLM Inferencecited by 89
  22. 2023ZeroQuant-V2: Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensationcited by 79
  23. 2023PanGu-Sigma: Towards Trillion Parameter Language Model with Sparse Heterogeneous Computingcited by 56
  24. 2020Masked Language Modeling for Proteins via Linearly Scalable Long-Context Transformerscited by 49
  25. 2021ERNIE 3.0 Titan: Exploring Larger-scale Knowledge Enhanced Pre-training for Language Understanding and Generationcited by 46
  26. 2023ZeroQuant-FP: A Leap Forward in LLMs Post-Training W4A8 Quantization Using Floating-Point Formatscited by 44
  27. 2022In-context Learning Distillation: Transferring Few-shot Learning Ability of Pre-trained Language Modelscited by 37
  28. 2023QuantEase: Optimization-based Quantization for Language Modelscited by 29
  29. 2020ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generatorscited by 27
  30. 2023Fewer is More: Boosting LLM Reasoning with Reinforced Context Pruningcited by 26
  31. 2024SparseLLM: Towards Global Pruning for Pre-trained Language Modelscited by 23
  32. 2023FineQuant: Unlocking Efficiency with Fine-Grained Weight-Only Quantization for LLMscited by 22
  33. 2022Measuring Datacited by 18
  34. 2023Efficient Prompting via Dynamic In-Context Learningcited by 18
  35. 2023TensorGPT: Efficient Compression of Large Language Models based on Tensor-Train Decompositioncited by 17
  36. 2024SelectLLM: Can LLMs Select Important Instructions to Annotate?cited by 16
  37. 2023Mini-GPTs: Efficient Large Language Models through Contextual Pruningcited by 6
  38. 2023DSFormer: Effective Compression of Text-Transformers by Dense-Sparse Weight Factorizationcited by 3

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 4; 2026: 1.

6 papers.

  1. 2025Alternatives To Next Token Prediction In Text Generation -- A Surveycited by 1
  2. 2024BEExformer: A Fast Inferencing Binarized Transformer with Early Exitscited by 0
  3. 2025Attention Condensation via Sparsity Induced Regularized Trainingcited by 0
  4. 2025Features extraction for image identification using computer visioncited by 0
  5. 2025Privacy-Preserving Offloading for Large Language Models in 6G Vehicular Networkscited by 0
  6. 2026Digitizing Coaching Intelligence: An Agentic Framework for Holistic Athlete Profiling using VLM and RAGcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Speed Always Wins: A Survey on Efficient Architectures for Large Language Models
  2. 2023A Quantitative Review on Language Model Efficiency Research
  3. 2026Transformer-Based Language Models Across Domain Verticals: Architectures, Applications and Critical Assessment
  4. 2024On Importance of Pruning and Distillation for Efficient Low Resource NLP
  5. 2024A critical review of methods and challenges in large language models
  6. 2026AE-LLM: Adaptive Efficiency Optimization for Large Language Models
  7. 2025From Large to Super-Tiny: End-to-End Optimization for Cost-Efficient LLMs
  8. 2023A Comprehensive Overview of Large Language Models