mlsanityData to 14 Oct 2026

Geometrically Principled Randomized Optimization for Efficient LLM Training

arXiv 2510.01878 ↗First posted 2 Oct 2025cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

Low-rank gradient optimization for large language models is currently divided into two categories: structured methods that rigorously identify subspaces, and randomized approaches employed primarily for computational efficiency. In this work, we question the intuition behind why random projections are effective. We trace this phenomenon to the geometry of the gradient subspaces, which exhibits subspace optimization landscape has a nearly flat curvature, while a significant portion of gradient information lies outside the core subspace. Leveraging these insights, and drawing on randomized linear algebra, we theoretically establish that random low-rank projections preserve the geometry, and we introduce GrassWalk and GrassJump, algorithms that navigate the Grassmannian manifold via random walks and jumps. By coupling this randomized exploration with subspace-aware optimizer and recovering the lost gradient signals, we achieve state-of-the-art results on LLaMA-1B, LLaMA-7B, and Qwen-1.5B pretraining. Our findings reframe randomization not merely as a computational shortcut, but as a geometrically principled approach to high-dimensional optimizations.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 32 of its 44 references that are in the map. First posted on arXiv 2 Oct 2025.

Builds on

32 of 44 references.

References · 32

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  3. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 463
  4. 2014Sketching as a Tool for Numerical Linear Algebracited by 288
  5. 2018Gradient Descent Happens in a Tiny Subspacecited by 217
  6. 2023ReLoRA: High-Rank Training Through Low-Rank Updatescited by 204
  7. 2024Adam-mini: Use Fewer Learning Rates To Gain Morecited by 137
  8. 2024Flora: Low-Rank Adapters Are Secretly Gradient Compressorscited by 108
  9. 2024LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuningcited by 89
  10. 2024Chain of LoRA: Efficient Fine-tuning of Language Models via Residual Learningcited by 82
  11. 2023Tied-Lora: Enhancing parameter efficiency of LoRA with weight tyingcited by 65
  12. 2024Fira: Can We Achieve Full-rank Training of LLMs Under Low-rank Constraint?cited by 43
20 more
  1. 2024Memory-Efficient LLM Training with Online Subspace Descentcited by 39
  2. 2024LDAdam: Adaptive Optimization from Low-Dimensional Gradient Statisticscited by 38
  3. 2024Does SGD really happen in tiny subspaces?cited by 36
  4. 2024Subspace Optimization for Large Language Models with Convergence Guaranteescited by 34
  5. 2024BAdam: A Memory Efficient Full Parameter Optimization Method for Large Language Modelscited by 27
  6. 2024Compressible Dynamics in Deep Overparameterized Low-Rank Learning & Adaptationcited by 26
  7. 2024MicroAdam: Accurate Adaptive Optimization with Low Space Overhead and Provable Convergencecited by 19
  8. 2024VeLoRA: Memory Efficient Training using Rank-1 Sub-Token Projectionscited by 18
  9. 2024Grass: Compute Efficient Low-Memory LLM Training with Structured Sparse Gradientscited by 18
  10. 2025SubTrack++ : Gradient Subspace Tracking for Scalable LLM Trainingcited by 16
  11. 2025A Memory Efficient Randomized Subspace Optimization Method for Training Large Language Modelscited by 16
  12. 2025Greedy Low-Rank Gradient Compression for Distributed Learning with Convergence Guaranteescited by 11
  13. 2024FRUGAL: Memory-Efficient Optimization by Reducing State Overhead for Scalable Trainingcited by 10
  14. 2024BlockLLM: Memory-Efficient Adaptation of LLMs by Selecting and Optimizing the Right Coordinate Blockscited by 9
  15. 2024Adapprox: Adaptive Approximation in Adam Optimization via Randomized Low-Rank Matricescited by 8
  16. 2024COAP: Memory-Efficient Training with Correlation-Aware Gradient Projectioncited by 8
  17. 2015Global Convergence of a Grassmannian Gradient Descent Algorithm for Subspace Estimationcited by 5
  18. 2024Identifying Policy Gradient Subspacescited by 5
  19. 2017Fast online low-rank tensor subspace tracking by CP decomposition using recursive least squares from incomplete observationscited by 4
  20. 2025Memory-Efficient Differentially Private Training with Gradient Random Projectioncited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026No Subspace to Track: Non-Identifiability and Optimizer State in Low-Rank Trainingcited by 0