mlsanityData to 14 Oct 2026

Computational Bottlenecks of Training Small-scale Large Language Models

arXiv 2410.19456 ↗First posted 25 Oct 2024cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

While large language models (LLMs) dominate the AI landscape, Small-scale large Language Models (SLMs) are gaining attention due to cost and efficiency demands from consumers. However, there is limited research on the training behavior and computational requirements of SLMs. In this study, we explore the computational bottlenecks of training SLMs (up to 2B parameters) by examining the effects of various hyperparameters and configurations, including GPU type, batch size, model size, communication protocol, attention type, and the number of GPUs. We assess these factors on popular cloud services using metrics such as loss per dollar and tokens per second. Our findings aim to support the broader adoption and optimization of language model training for low-resource AI research institutes.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 25 of its 26 references that are in the map. First posted on arXiv 25 Oct 2024.

Builds on

25 of 26 references.

References · 25

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2024The Llama 3 Herd of Modelscited by 14,715
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  7. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  8. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  9. 2020Scaling Laws for Neural Language Modelscited by 6,586
  10. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  11. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  12. 2023Mistral 7Bcited by 3,164
13 more
  1. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  2. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  3. 2020Knowledge Distillation: A Surveycited by 1,603
  4. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 660
  5. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 634
  6. 2024TinyLlama: An Open-Source Small Language Modelcited by 618
  7. 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 527
  8. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 415
  9. 2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 385
  10. 2024MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Casescited by 223
  11. 2024Compact Language Models via Pruning and Knowledge Distillationcited by 148
  12. 2023Full Stack Optimization of Transformer Inference: a Surveycited by 80
  13. 2024Smaller, Weaker, Yet Better: Training LLM Reasoners via Compute-Optimal Samplingcited by 61

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 1.

2 papers.

  1. 2024BlueLM-V-3B: Algorithm and System Co-Design for Multimodal Large Language Models on Mobile Devicescited by 25
  2. 2025Towards a Small Language Model Lifecycle Frameworkcited by 0