mlsanityData to 7 Oct 2026

Efficient Methods for Natural Language Processing: A Survey

arXiv 2209.00099 ↗First posted 31 Aug 2022cs.CLcited by 78

Not on the map. Why?

Abstract (arXiv)

Recent work in natural language processing (NLP) has yielded appealing results from scaling model parameters and training data; however, using only scale to improve performance means that resource consumption also grows. Such resources include data, time, storage, or energy, all of which are naturally limited and unevenly distributed. This motivates research into efficient methods that require fewer resources to achieve similar results. This survey synthesizes and relates current methods and findings in efficient NLP. We aim to provide both guidance for conducting NLP under limited resources, and point towards promising research directions for developing more efficient methods.

Key facts. Cited by 78 papers in the map (data through 7 Oct 2026). Builds on 234 of its 330 references that are in the map. First posted on arXiv 31 Aug 2022.

Builds on

234 of 330 references.

References · 234

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  8. 2020Scaling Laws for Neural Language Modelscited by 6,411
  9. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  10. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  11. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  12. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
38 more
  1. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  2. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  3. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  4. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  5. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  6. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  7. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  8. 2020Longformer: The Long-Document Transformercited by 3,362
  9. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  10. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  11. 2022Emergent Abilities of Large Language Modelscited by 2,483
  12. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  13. 2018Deep contextualized word representationscited by 2,285
  14. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  15. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  16. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  17. 2019Language Models as Knowledge Bases?cited by 1,922
  18. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  19. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  20. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  21. 2020Reformer: The Efficient Transformercited by 1,658
  22. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  23. 2020Rethinking Attention with Performerscited by 1,584
  24. 2020Knowledge Distillation: A Surveycited by 1,578
  25. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  26. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  27. 2018Datasheets for Datasetscited by 1,371
  28. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  29. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  30. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  31. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,219
  32. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  33. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  34. 2017Mixed Precision Trainingcited by 1,135
  35. 2018Self-Attention with Relative Position Representationscited by 1,112
  36. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  37. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  38. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890

50 most cited of 234 All 234 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 29; 2024: 24; 2025: 16; 2026: 5.

78 papers.

Sort:

By area:

  1. 2024Hybrid LLM: Cost-Efficient and Quality-Aware Query Routingcited by 354
  2. 2024Aya Model: An Instruction Finetuned Open-Access Multilingual Language Modelcited by 292
  3. 2024Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inferencecited by 115
  4. 2023Parameter-Efficient Orthogonal Finetuning via Butterfly Factorizationcited by 112
  5. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 101
  6. 2023Uncovering mesa-optimization algorithms in Transformerscited by 94
  7. 2023SkipDecode: Autoregressive Skip Decoding with Batching and Caching for Efficient LLM Inferencecited by 89
  8. 2023Red teaming ChatGPT via Jailbreaking: Bias, Robustness, Reliability and Toxicitycited by 86
  9. 2023Cascade Speculative Drafting for Even Faster LLM Inferencecited by 83
  10. 2022Cramming: Training a Language Model on a Single GPU in One Daycited by 81
  11. 2025BEST-Route: Adaptive LLM Routing with Test-Time Optimal Computecited by 63
  12. 2023Active Learning Principles for In-Context Learning with Large Language Modelscited by 53
38 more
  1. 2023On the Impact of Calibration Data in Post-training Quantization and Pruningcited by 51
  2. 2023Intriguing Properties of Quantization at Scalecited by 48
  3. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 46
  4. 2024The Larger the Better? Improved LLM Code-Generation via Budget Reallocationcited by 41
  5. 2023Exploring Parameter-Efficient Fine-Tuning Techniques for Code Generation with Large Language Modelscited by 28
  6. 2023On Efficient Training of Large-Scale Deep Learning Models: A Literature Reviewcited by 26
  7. 2023Omni-SMoLA: Boosting Generalist Multimodal Models with Soft Mixture of Low-rank Expertscited by 26
  8. 2023The Efficiency Spectrum of Large Language Models: An Algorithmic Surveycited by 25
  9. 2023Model Compression in Practice: Lessons Learned from Practitioners Creating On-device Machine Learning Experiencescited by 21
  10. 2022A Survey for Efficient Open Domain Question Answeringcited by 20
  11. 2024Improving Model Evaluation using SMART Filtering of Benchmark Datasetscited by 20
  12. 2024Comparison of Static Application Security Testing Tools and Large Language Models for Repo-level Vulnerability Detectioncited by 18
  13. 2023Domain Terminology Integration into Machine Translation: Leveraging Large Language Modelscited by 17
  14. 2024ActiveLLM: Large Language Model-based Active Learning for Textual Few-Shot Scenarioscited by 17
  15. 2024On the Limitations of Compute Thresholds as a Governance Strategycited by 17
  16. 2023A Systematic Study of Knowledge Distillation for Natural Language Generation with Pseudo-Target Trainingcited by 16
  17. 2024MT2ST: Adaptive Multi-Task to Single-Task Learningcited by 16
  18. 2022Intriguing Properties of Compression on Multilingual Modelscited by 15
  19. 2025AdaptiVocab: Enhancing LLM Efficiency in Focused Domains through Lightweight Vocabulary Adaptationcited by 15
  20. 2024AutoRAG-HP: Automatic Online Hyper-Parameter Tuning for Retrieval-Augmented Generationcited by 13
  21. 2023Optical Transformerscited by 10
  22. 2024Self-calibration for Language Model Quantization and Pruningcited by 10
  23. 2025The Multilingual Divide and Its Impact on Global AI Safetycited by 10
  24. 2024Natural Language Processing RELIES on Linguisticscited by 9
  25. 2025ML-SpecQD: Multi-Level Speculative Decoding with Quantized Draftscited by 9
  26. 2023Neural Natural Language Processing for Long Texts: A Survey on Classification and Summarizationcited by 7
  27. 2023On "Scientific Debt" in NLP: A Case for More Rigour in Language Model Pre-Training Researchcited by 7
  28. 2023Efficiency Pentathlon: A Standardized Arena for Efficiency Evaluationcited by 7
  29. 2024Nexus: Specialization meets Adaptability for Efficiently Training Mixture of Expertscited by 7
  30. 2023Finding the SWEET Spot: Analysis and Improvement of Adaptive Inference in Low Resource Settingscited by 5
  31. 2023Surveying (Dis)Parities and Concerns of Compute Hungry NLP Researchcited by 5
  32. 2024SWAT: Scalable and Efficient Window Attention-based Transformers Acceleration on FPGAscited by 5
  33. 2024Compress and Compare: Interactively Evaluating Efficiency and Behavior Across ML Model Compression Experimentscited by 4
  34. 2024Fine-tuning Large Language Models with Limited Data: A Survey and Practical Guidecited by 4
  35. 2025Efficient Speech Translation through Model Compression and Knowledge Distillationcited by 4
  36. 2025Iterative Layer Pruning for Efficient Translation Inferencecited by 4
  37. 2024Enhancing Dialogue Generation in Werewolf Game Through Situation Analysis and Persuasion Strategiescited by 3
  38. 2025Enterprise AI Must Enforce Participant-Aware Access Controlcited by 3

50 most cited of 78 All 78 (JSON)