mlsanityData to 14 Oct 2026

LaX: Boosting Low-Rank Training of Foundation Models via Latent Crossing

arXiv 2505.21732 ↗First posted 27 May 2025cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

Training foundation models such as ViTs and LLMs requires tremendous computing cost. Low-rank matrix or tensor factorization offers a parameter-efficient alternative, but often downgrades performance due to the restricted parameter space. In this work, we introduce Latent Crossing (LaX) -- a simple yet effective plug-and-play module that enhances the capacity of low-rank models by enabling information flow across low-rank subspaces. We extensively validate the benefits of LaX on pre-training tasks with ViT-Base/Large and LLaMA-like models ranging from 60M to 1B parameters. LaX boosts low-rank model performance to match or exceed the full-rank baselines while using 2-3(×) fewer parameters. When equipped with low-rank adapters (i.e., LoRA) for fine-tuning LLaMA-7/13B, LaX consistently improves performance on arithmetic and common sense reasoning tasks with negligible cost.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 44 of its 67 references that are in the map. First posted on arXiv 27 May 2025.

Builds on

44 of 67 references.

References · 44

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  6. 2024The Llama 3 Herd of Modelscited by 14,715
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  8. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  9. 2020Scaling Laws for Neural Language Modelscited by 6,586
  10. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  11. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,546
  12. 2016Identity Mappings in Deep Residual Networkscited by 1,965
32 more
  1. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 758
  2. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 694
  3. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 463
  4. 2018Bag of Tricks for Image Classification with Convolutional Neural Networkscited by 431
  5. 2024LoRA+: Efficient Low Rank Adaptation of Large Modelscited by 377
  6. 2023LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Modelscited by 324
  7. 2014Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluationcited by 306
  8. 2014Speeding up Convolutional Neural Networks with Low Rank Expansionscited by 284
  9. 2015Tensorizing Neural Networkscited by 250
  10. 2023ReLoRA: High-Rank Training Through Low-Rank Updatescited by 204
  11. 2014Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decompositioncited by 203
  12. 2015Compression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applicationscited by 162
  13. 2023LoRA-FA: Efficient and Effective Low Rank Representation Fine-tuningcited by 137
  14. 2016Tensor Ring Decompositioncited by 127
  15. 2015Accelerating Very Deep Convolutional Networks for Classification and Detectioncited by 109
  16. 2024Scaling Laws for Precisioncited by 95
  17. 2016Ultimate tensorization: compressing convolutional and FC layers alikecited by 56
  18. 2018Wide Compression: Tensor Ring Netscited by 50
  19. 2022Rank Diminishing in Deep Neural Networkscited by 47
  20. 2024SLTrain: a sparse plus low-rank approach for parameter and memory efficient pretrainingcited by 45
  21. 2021Compressing Neural Networks: Towards Determining the Optimal Layer-wise Decompositioncited by 33
  22. 2022Exploring Low Rank Training of Deep Neural Networkscited by 30
  23. 2021Enabling Lightweight Fine-tuning for Pre-trained Language Model Compression based on Matrix Product Operatorscited by 29
  24. 2025CoLA: Compute-Efficient Pre-Training of LLMs via Low-Rank Activationcited by 28
  25. 2020Hybrid Tensor Decomposition in Neural Network Compressioncited by 15
  26. 2024CoMERA: Computing- and Memory-Efficient Training via Rank-Adaptive Tensor Optimizationcited by 15
  27. 2024Residual Alignment: Uncovering the Mechanisms of Residual Networkscited by 13
  28. 2023How Sparse Can We Prune A Deep Network: A Fundamental Limit Perspectivecited by 9
  29. 2025An Overview of Low-Rank Structures in the Training and Adaptation of Large Modelscited by 9
  30. 2021Diffusion Mechanism in Residual Neural Network: Theory and Applicationscited by 5
  31. 2023Low Rank Optimization for Efficient Deep Learning: Making A Balance between Compact Architecture and Fast Trainingcited by 5
  32. 2023Causal Deep Learningcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 5.

6 papers.

  1. 2026Muon2: Boosting Muon via Adaptive Second-Moment Preconditioningcited by 17
  2. 2026TEON: Tensorized Orthonormalization Beyond Layer-Wise Muon for Large Language Model Pre-Trainingcited by 6
  3. 2026MUON+: Towards More Effective Muon via One Additional Normalization Step for LLM Pre-trainingcited by 6
  4. 2026Synergistic Intra- and Cross-Layer Regularization Losses for MoE Expert Specializationcited by 5
  5. 2025BOOST: BOttleneck-Optimized Scalable Training Framework for Low-Rank Large Language Modelscited by 2
  6. 2026GRZO: Group-Relative Zeroth-Order Optimization for Large Language Model Fine-Tuningcited by 0