mlsanityData to 7 Oct 2026

STEP: Staged Parameter-Efficient Pre-training for Large Language Models

arXiv 2504.04151 ↗First posted 5 Apr 2025cs.CLcited by 2

Not on the map. Why?

Abstract (arXiv)

Pre-training large language models (LLMs) faces significant memory challenges due to the large size of model parameters. We introduce STaged parameter-Efficient Pre-training (STEP), which integrates parameter-efficient tuning techniques with model growth. We conduct experiments on pre-training LLMs of various sizes and demonstrate that STEP achieves up to a 53.9% reduction in maximum memory requirements compared to vanilla pre-training while maintaining equivalent performance. Furthermore, we show that the model by STEP performs comparably to vanilla pre-trained models on downstream tasks after instruction tuning.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 32 of its 42 references that are in the map. First posted on arXiv 5 Apr 2025.

Builds on

32 of 42 references.

References · 32

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  7. 2020Scaling Laws for Neural Language Modelscited by 6,411
  8. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  9. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  10. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  11. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  12. 2016Pointer Sentinel Mixture Modelscited by 2,992
20 more
  1. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  2. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  3. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  4. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  5. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  6. 2017Mixed Precision Trainingcited by 1,135
  7. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
  8. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  9. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  10. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 447
  11. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 300
  12. 2023ReLoRA: High-Rank Training Through Low-Rank Updatescited by 200
  13. 2024LLaMA Pro: Progressive LLaMA with Block Expansioncited by 99
  14. 2017Multi-level Residual Networks from Dynamical Systems Viewcited by 59
  15. 2022Staged Training for Transformer Language Modelscited by 49
  16. 2022Automated Progressive Learning for Efficient Training of Vision Transformerscited by 36
  17. 2023Masked Structural Growth for 2x Faster Language Model Pre-trainingcited by 32
  18. 2023Is Bigger and Deeper Always Better? Probing LLaMA Across Scales and Layerscited by 20
  19. 2023Alleviating Over-smoothing for Unsupervised Sentence Representationcited by 16
  20. 2024Preparing Lessons for Progressive Training on Language Modelscited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025TopK Language Modelscited by 0
  2. 2026Adapting Text LLMs to Speech via Multimodal Depth Up-Scalingcited by 0