mlsanityData to 14 Oct 2026

Breaking the Frozen Subspace: Importance Sampling for Low-Rank Optimization in LLM Pretraining

arXiv 2502.05790 ↗First posted 9 Feb 2025cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

Low-rank optimization has emerged as a promising approach to enabling memory-efficient training of large language models (LLMs). Existing low-rank optimization methods typically project gradients onto a low-rank subspace, reducing the memory cost of storing optimizer states. A key challenge in these methods is selecting suitable subspaces to ensure an effective optimization trajectory. Most existing approaches select the dominant subspace to preserve gradient information, as this intuitively provides the best approximation. However, we find that in practice, the dominant subspace stops changing during pretraining, thereby constraining weight updates to similar subspaces. In this paper, we propose importance sampling for low-rank optimization in LLM pretraining with a provable convergence guarantee, which the dominant subspace approach does not have. Empirically, we demonstrate that our method significantly outperforms previous methods in LLM pretraining tasks.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 64 of its 83 references that are in the map. First posted on arXiv 9 Feb 2025.

Builds on

64 of 83 references.

References · 64

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2022Training language models to follow instructions with human feedbackcited by 15,971
  5. 2024The Llama 3 Herd of Modelscited by 14,715
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  7. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  8. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  10. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  11. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  12. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
38 more
  1. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  2. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 463
  3. 20218-bit Optimizers via Block-wise Quantizationcited by 403
  4. 2018Gradient Descent Happens in a Tiny Subspacecited by 217
  5. 2023ReLoRA: High-Rank Training Through Low-Rank Updatescited by 204
  6. 2023TEMPO: Prompt-based Generative Pre-trained Transformer for Time Series Forecastingcited by 174
  7. 2024Adam-mini: Use Fewer Learning Rates To Gain Morecited by 137
  8. 2024Flora: Low-Rank Adapters Are Secretly Gradient Compressorscited by 108
  9. 2024LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuningcited by 89
  10. 2020Sample Efficient Reinforcement Learning with REINFORCEcited by 86
  11. 2024Chain of LoRA: Efficient Fine-tuning of Language Models via Residual Learningcited by 82
  12. 2023Delta-LoRA: Fine-Tuning High-Rank Parameters with the Delta of Low-Rank Matricescited by 74
  13. 2023Memory Efficient Optimizers with 4-bit Statescited by 54
  14. 2024SLTrain: a sparse plus low-rank approach for parameter and memory efficient pretrainingcited by 45
  15. 2024Fira: Can We Achieve Full-rank Training of LLMs Under Low-rank Constraint?cited by 43
  16. 2024Memory-Efficient LLM Training with Online Subspace Descentcited by 39
  17. 2024Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradientscited by 38
  18. 2023CAME: Confidence-guided Adaptive Memory Efficient Optimizationcited by 36
  19. 2024Does SGD really happen in tiny subspaces?cited by 36
  20. 2023An Iterative Algorithm for Rescaled Hyperbolic Functions Regressioncited by 35
  21. 2023MosaicBERT: A Bidirectional Encoder Optimized for Fast Pretrainingcited by 34
  22. 2024Subspace Optimization for Large Language Models with Convergence Guaranteescited by 34
  23. 2023In-Context Learning for Attention Scheme: from Single Softmax Regression to Multiple Softmax Regression via a Tensor Trickcited by 32
  24. 2024Conv-Basis: A New Paradigm for Efficient Attention Inference and Gradient Computation in Transformerscited by 32
  25. 2023Low Rank Matrix Completion via Robust Alternating Minimization in Nearly Linear Timecited by 31
  26. 2023A Mathematical Abstraction for Balancing the Trade-off Between Creativity and Reality in Large Language Modelscited by 29
  27. 2024QueryAgent: A Reliable and Efficient Reasoning Framework with Environmental Feedback-based Self-Correctioncited by 27
  28. 2023GradientCoin: A Peer-to-Peer Decentralized Large Language Modelscited by 23
  29. 2024H-STAR: LLM-driven Hybrid SQL-Text Adaptive Reasoning on Tablescited by 21
  30. 2024The Computational Limits of State-Space Models and Mamba via the Lens of Circuit Complexitycited by 21
  31. 2023Solving Attention Kernel Regression Problem via Pre-conditionercited by 20
  32. 2024Tree-of-Table: Unleashing the Power of LLMs for Enhanced Large-Scale Table Understandingcited by 20
  33. 2024PeriodicLoRA: Breaking the Low-Rank Bottleneck in LoRA Optimizationcited by 19
  34. 2024HFT: Half Fine-Tuning for Large Language Modelscited by 18
  35. 2024Grass: Compute Efficient Low-Memory LLM Training with Structured Sparse Gradientscited by 18
  36. 2024RoPE Attention Can Be Trained in Almost Linear Timecited by 13
  37. 2023A Unified Scheme of ResNet and Softmaxcited by 12
  38. 2019Stochastic Subspace Descentcited by 10

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 4.

7 papers.

  1. 202570% Size, 100% Accuracy: Lossless LLM Compression for Efficient GPU Inference via Dynamic-Length Float (DFloat11)cited by 24
  2. 2025Scalable Parameter and Memory Efficient Pretraining for LLM: Recent Algorithmic Advances and Benchmarkingcited by 5
  3. 2025Memory-Efficient Fine-Tuning via Low-Rank Activation Compressioncited by 2
  4. 2026From O(mn) to O(r2): Two-Sided Low-Rank Communication for Adam in Distributed Training with Memory Efficiencycited by 0
  5. 2026BROS: Bias-Corrected Randomized Subspaces for Memory-Efficient Single-Loop Bilevel Optimizationcited by 0
  6. 2026MoARa: Module-Aware Rank Allocation and Structure-Preserving Decomposition for Low-Rank LLM Pre-trainingcited by 0
  7. 2026Memory-Efficient Subspace Optimization with Residual Correction: From Momentum SGD to Muoncited by 0