mlsanityData to 7 Oct 2026

Efficient Pre-Training of LLMs through Truncated SVD Representations

arXiv 2605.28573 ↗First posted 27 May 2026cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

LLM pretraining is extremely costly; therefore, parameter-efficient LLM architectures have recently emerged as a compelling research direction. One such promising approach is to represent the parameters as orthonormal low-rank weight matrices. However, maintaining orthonormality during training is computationally expensive, making it impractical. This paper presents the TSVD (Truncated Singular Value Decomposition) framework which efficiently maintains orthonormality through QR decomposition and caching. Furthermore, a spectral energy heuristic is introduced to select the rank of the resulting low-rank weight matrices. Empirical evaluations across model sizes show that TSVD matches or outperforms full-parameter baselines at a fraction of the compute cost. TSVD thus provides a scalable, computationally efficient foundation for LLM pretraining.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 30 of its 40 references that are in the map. First posted on arXiv 27 May 2026.

Builds on

30 of 40 references.

References · 30

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2020Scaling Laws for Neural Language Modelscited by 6,411
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  7. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  8. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  9. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  10. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  11. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 481
  12. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 447
18 more
  1. 2015Unitary Evolution Recurrent Neural Networkscited by 291
  2. 2023ReLoRA: High-Rank Training Through Low-Rank Updatescited by 200
  3. 2017Orthogonal Weight Normalization: Solution to Optimization over Multiple Dependent Stiefel Manifolds in Deep Neural Networkscited by 92
  4. 2018Can We Gain More from Orthogonality Regularizations in Training Deep CNNs?cited by 62
  5. 2021Initialization and Regularization of Factorized Neural Layerscited by 58
  6. 2020Learning Low-rank Deep Neural Networks via Singular Vector Orthogonality Regularization and Singular Value Sparsificationcited by 52
  7. 2024SLTrain: a sparse plus low-rank approach for parameter and memory efficient pretrainingcited by 44
  8. 2022Exploring Low Rank Training of Deep Neural Networkscited by 30
  9. 2025CoLA: Compute-Efficient Pre-Training of LLMs via Low-Rank Activationcited by 28
  10. 2024CompAct: Compressed Activations for Memory-Efficient LLM Trainingcited by 15
  11. 2025StelLA: Subspace Learning in Low-rank Adaptation using Stiefel Manifoldcited by 13
  12. 2023Robust low-rank training via approximate orthonormal constraintscited by 12
  13. 2024ELRT: Efficient Low-Rank Training for Compact Convolutional Neural Networkscited by 11
  14. 2025LOST: Low-rank and Sparse Pre-training for Large Language Modelscited by 10
  15. 2025Reparameterized LLM Training via Orthogonal Equivalence Transformationcited by 8
  16. 2024Investigating Low-Rank Training in Transformer Language Models: Efficiency and Scaling Analysiscited by 6
  17. 2024Full-Rank No More: Low-Rank Weight Training for Modern Speech Recognition Modelscited by 3
  18. 2026POET-X: Memory-efficient LLM Training by Scaling Orthogonal Transformationcited by 3

Built on by

Not cited yet.