mlsanityData to 7 Oct 2026

The Cost of Training NLP Models: A Concise Overview

arXiv 2004.08900 ↗First posted 19 Apr 2020cs.CL cs.LG cs.NEcited by 97

Not on the map. Why?

Abstract (arXiv)

We review the cost of training large-scale language models, and the drivers of these costs. The intended audience includes engineers and scientists budgeting their model-training experiments, as well as non-practitioners trying to make sense of the economics of modern-day Natural Language Processing (NLP).

Key facts. Cited by 97 papers in the map (data through 7 Oct 2026). First posted on arXiv 19 Apr 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 8; 2022: 27; 2023: 30; 2024: 21; 2025: 9; 2026: 2.

97 papers.

Sort:

By area:

  1. 2022Confident Adaptive Language Modelingcited by 297
  2. 2022VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidancecited by 285
  3. 2024Knowledge Conflicts for LLMs: A Surveycited by 274
  4. 2021GRAD-MATCH: Gradient Matching based Data Subset Selection for Efficient Deep Model Trainingcited by 269
  5. 2022Tutel: Adaptive Mixture-of-Experts at Scalecited by 199
  6. 2022Compute Trends Across Three Eras of Machine Learningcited by 161
  7. 2022MRKL Systems: A modular, neuro-symbolic architecture that combines large language models, external knowledge sources and discrete reasoningcited by 153
  8. 2021The Efficiency Misnomercited by 98
  9. 2022Deep Unsupervised Domain Adaptation: A Review of Recent Advances and Perspectivescited by 98
  10. 2023Evaluating the Social Impact of Generative AI Systems in Systems and Societycited by 91
  11. 2024BioMedLM: A 2.7B Parameter Language Model Trained On Biomedical Textcited by 85
  12. 2021Consistent Accelerated Inference via Confident Adaptive Transformerscited by 73
38 more
  1. 2024A Sober Look at LLMs for Material Discovery: Are They Actually Good for Bayesian Optimization Over Molecules?cited by 46
  2. 2024Adaptive Data Optimization: Dynamic Sample Selection with Scaling Lawscited by 46
  3. 2025Comet: Fine-grained Computation-communication Overlapping for Mixture-of-Expertscited by 46
  4. 2022Machine Learning Model Sizes and the Parameter Gapcited by 39
  5. 2022Compute-Efficient Deep Learning: Algorithmic Trends and Opportunitiescited by 38
  6. 2022Is Neuro-Symbolic AI Meeting its Promise in Natural Language Processing? A Structured Reviewcited by 37
  7. 2022Dataset Inference for Self-Supervised Modelscited by 36
  8. 2023DoWG Unleashed: An Efficient Universal Parameter-Free Gradient Descent Methodcited by 35
  9. 2023Lost in Translation: Large Language Models in Non-English Content Analysiscited by 34
  10. 2022Standing on the Shoulders of Giant Frozen Language Modelscited by 33
  11. 2021On the Existence of Universal Lottery Ticketscited by 30
  12. 2022Green Learning: Introduction, Examples and Outlookcited by 30
  13. 2023The Shaky Foundations of Clinical Foundation Models: A Survey of Large Language Models and Foundation Models for EMRscited by 29
  14. 2021Dynamic Language Models for Continuously Evolving Contentcited by 27
  15. 2023MoMo: Momentum Models for Adaptive Learning Ratescited by 24
  16. 2022On the Difficulty of Defending Self-Supervised Learning against Model Extractioncited by 22
  17. 2023An Overview on Language Models: Recent Developments and Outlookcited by 22
  18. 2024TAPT: Test-Time Adversarial Prompt Tuning for Robust Inference in Vision-Language Modelscited by 22
  19. 2023On Optimal Caching and Model Multiplexing for Large Model Inferencecited by 20
  20. 2024The Compute Divide in Machine Learning: A Threat to Academic Contribution and Scrutiny?cited by 20
  21. 2023Improving Retrieval-Augmented Large Language Models via Data Importance Learningcited by 17
  22. 2022Mokey: Enabling Narrow Fixed-Point Inference for Out-of-the-Box Floating-Point Transformer Modelscited by 16
  23. 2024Optimizing Language Model's Reasoning Abilities with Weak Supervisioncited by 15
  24. 2024Towards Understanding Systems Trade-offs in Retrieval-Augmented Generation Model Inferencecited by 15
  25. 2021Perspectives and Prospects on Transformer Architecture for Cross-Modal Tasks with Language and Visioncited by 14
  26. 2021Automated Essay Scoring Using Transformer Modelscited by 14
  27. 2023Provable Data Subset Selection For Efficient Neural Network Trainingcited by 14
  28. 2023Automatic Gradient Descent: Deep Learning without Hyperparameterscited by 14
  29. 2022PASHA: Efficient HPO and NAS with Progressive Resource Allocationcited by 13
  30. 2022BudgetLongformer: Can we Cheaply Pretrain a SotA Legal Language Model From Scratch?cited by 13
  31. 2022DCT-Former: Efficient Self-Attention with Discrete Cosine Transformcited by 12
  32. 2024Analyzing the Role of Semantic Representations in the Era of Large Language Modelscited by 12
  33. 2022A survey on knowledge-enhanced multimodal learningcited by 11
  34. 2021GIST: Distributed Training for Large-Scale Graph Convolutional Networkscited by 10
  35. 2024Understanding Adam Requires Better Rotation Dependent Assumptionscited by 10
  36. 2024Efficient Prompt Caching via Embedding Similaritycited by 9
  37. 2024Gradient descent with generalized Newton's methodcited by 9
  38. 2023Using Artificial Populations to Study Psychological Phenomena in Neural Modelscited by 8

50 most cited of 97 All 97 (JSON)