mlsanityData to 7 Oct 2026

A Survey on Memory-Efficient Transformer-Based Model Training in AI for Science

arXiv 2501.11847 ↗First posted 21 Jan 2025cs.LG cs.AIcited by 3

Not on the map. Why?

Abstract (arXiv)

Scientific research faces high costs and inefficiencies with traditional methods, but the rise of deep learning and large language models (LLMs) offers innovative solutions. This survey reviews transformer-based LLM applications across scientific fields such as biology, medicine, chemistry, and meteorology, underscoring their role in advancing research. However, the continuous expansion of model size has led to significant memory demands, hindering further development and application of LLMs for science. This survey systematically reviews and categorizes memory-efficient pre-training techniques for large-scale transformers, including algorithm-level, system-level, and hardware-software co-optimization. Using AlphaFold 2 as an example, we demonstrate how tailored memory optimization methods can reduce storage needs while preserving prediction accuracy. By bridging model efficiency and scientific application needs, we hope to provide insights for scalable and cost-effective LLM training in AI for science.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 69 of its 112 references that are in the map. First posted on arXiv 21 Jan 2025.

Builds on

69 of 112 references.

References · 69

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2022Training language models to follow instructions with human feedbackcited by 15,712
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  8. 2020Scaling Laws for Neural Language Modelscited by 6,411
  9. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  10. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  11. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  12. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
38 more
  1. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  2. 2024Qwen2 Technical Reportcited by 2,076
  3. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  4. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  5. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  6. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  7. 2020Rethinking Attention with Performerscited by 1,584
  8. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  9. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  10. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,215
  11. 2017Mixed Precision Trainingcited by 1,135
  12. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  13. 2016Training Deep Nets with Sublinear Memory Costcited by 825
  14. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  15. 2024Yi: Open Foundation Models by 01.AIcited by 706
  16. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 641
  17. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  18. 2020SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networkscited by 602
  19. 2022BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Miningcited by 531
  20. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 507
  21. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 448
  22. 2023Symbolic Discovery of Optimization Algorithmscited by 423
  23. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 339
  24. 2023Fine-Tuning Language Models with Just Forward Passescited by 337
  25. 2023ClimaX: A foundation model for weather and climatecited by 325
  26. 2023LLM-QAT: Data-Free Quantization Aware Training for Large Language Modelscited by 320
  27. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 300
  28. 2024The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bitscited by 289
  29. 2023Towards Generalist Biomedical AIcited by 286
  30. 2023MEDITRON-70B: Scaling Medical Pretraining for Large Language Modelscited by 272
  31. 2021ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learningcited by 253
  32. 2023FuXi: A cascade machine learning forecasting system for 15-day global weather forecastcited by 237
  33. 2019A Study of BFLOAT16 for Deep Learning Trainingcited by 227
  34. 2023HuatuoGPT, towards Taming Language Model to Be a Doctorcited by 211
  35. 2023BiomedGPT: A Generalist Vision-Language Foundation Model for Diverse Biomedical Taskscited by 201
  36. 2023BitNet: Scaling 1-bit Transformers for Large Language Modelscited by 175
  37. 2022Pangu-Weather: A 3D High-Resolution Model for Fast and Accurate Global Weather Forecastcited by 164
  38. 2021Self-attention Does Not Need O(n2) Memorycited by 152

50 most cited of 69 All 69 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025Preconditioned Norms: A Unified Framework for Steepest Descent, Quasi-Newton and Adaptive Methodscited by 9
  2. 2026GRASS: Gradient-based Adaptive Layer-wise Importance Sampling for Memory-efficient Large Language Model Fine-tuningcited by 1
  3. 2026Memory-Efficient Activation Checkpointing with Sliding Window and Hirschberg's Algorithm for 0/1 Knapsack Solving in PyTorchcited by 0