mlsanityData to 7 Oct 2026

Training Neural Networks from Scratch with Parallel Low-Rank Adapters

arXiv 2402.16828 ↗First posted 26 Feb 2024cs.LG cs.AI cs.CVcited by 17

Not on the map. Why?

Abstract (arXiv)

The scalability of deep learning models is fundamentally limited by computing resources, memory, and communication. Although methods like low-rank adaptation (LoRA) have reduced the cost of model finetuning, its application in model pre-training remains largely unexplored. This paper explores extending LoRA to model pre-training, identifying the inherent constraints and limitations of standard LoRA in this context. We introduce LoRA-the-Explorer (LTE), a novel bi-level optimization algorithm designed to enable parallel training of multiple low-rank heads across computing nodes, thereby reducing the need for frequent synchronization. Our approach includes extensive experimentation on vision transformers using various vision datasets, demonstrating that LTE is competitive with standard pre-training.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Builds on 79 of its 111 references that are in the map. First posted on arXiv 26 Feb 2024.

Builds on

79 of 111 references.

References · 79

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  5. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  6. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  7. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  8. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  9. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  10. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  11. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  12. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,339
38 more
  1. 2019Contrastive Multiview Codingcited by 1,297
  2. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,220
  3. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  4. 2022Editing Models with Task Arithmeticcited by 1,076
  5. 2020Adaptive Federated Optimizationcited by 1,036
  6. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,021
  7. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 836
  8. 2016Training Deep Nets with Sublinear Memory Costcited by 825
  9. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 795
  10. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627
  11. 2021Merging Models with Fisher-Weighted Averagingcited by 613
  12. 2018Local SGD Converges Fast and Communicates Littlecited by 595
  13. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 585
  14. 2022Vision Transformer Adapter for Dense Predictionscited by 564
  15. 2017Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descentcited by 496
  16. 2017Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Trainingcited by 471
  17. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 460
  18. 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modelingcited by 429
  19. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 425
  20. 2020A Unified Theory of Decentralized SGD with Changing Topology and Local Updatescited by 361
  21. 2017Snapshot Ensembles: Train 1, get M for freecited by 349
  22. 2018Essentially No Barriers in Neural Network Energy Landscapecited by 328
  23. 2023DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Datacited by 322
  24. 2021VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Taskscited by 319
  25. 2021A Field Guide to Federated Optimizationcited by 312
  26. 2021The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networkscited by 285
  27. 2018Parallel Restarted SGD with Faster Convergence and Less Communication: Demystifying Why Model Averaging Works for Deep Learningcited by 277
  28. 2018An Empirical Model of Large-Batch Trainingcited by 268
  29. 2017Sparse Communication for Distributed Gradient Descentcited by 243
  30. 2014Understanding image representations by measuring their equivariance and equivalencecited by 240
  31. 2019Uniform convergence may be unable to explain generalization in deep learningcited by 223
  32. 2019Decentralized Stochastic Optimization and Gossip Algorithms with Compressed Communicationcited by 218
  33. 2017The Reversible Residual Network: Backpropagation Without Storing Activationscited by 215
  34. 2021Revisiting Model Stitching to Compare Neural Representationscited by 203
  35. 2018Don't Use Large Mini-Batches, Use Local SGDcited by 201
  36. 2018Measuring the Effects of Data Parallelism on Neural Network Trainingcited by 190
  37. 2020Feature Learning in Infinite-Width Neural Networkscited by 182
  38. 2023ZipIt! Merging Models from Different Tasks without Trainingcited by 178

50 most cited of 79 All 79 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 6; 2026: 3.

17 papers.

Sort:

By area:

  1. 2024A Survey on LoRA of Large Language Modelscited by 67
  2. 2024A Survey on Model MoErging: Recycling and Routing Among Specialized Experts for Collaborative Learningcited by 52
  3. 2024SLTrain: a sparse plus low-rank approach for parameter and memory efficient pretrainingcited by 44
  4. 2025CoLA: Compute-Efficient Pre-Training of LLMs via Low-Rank Activationcited by 28
  5. 2025Communication-Efficient Language Model Training Scales Reliably and Robustly: Scaling Laws for DiLoCocited by 23
  6. 2024CoMERA: Computing- and Memory-Efficient Training via Rank-Adaptive Tensor Optimizationcited by 14
  7. 2024Fine-Tuning Attention Modules Only: Enhancing Weight Disentanglement in Task Arithmeticcited by 14
  8. 2024Pareto Low-Rank Adapters: Efficient Multi-Task Learning with Preferencescited by 13
  9. 2025Reparameterized LLM Training via Orthogonal Equivalence Transformationcited by 8
  10. 2026Stabilizing Native Low-Rank LLM Pretrainingcited by 6
  11. 2025One Rank at a Time: Cascading Error Dynamics in Sequential Learningcited by 3
  12. 2026POET-X: Memory-efficient LLM Training by Scaling Orthogonal Transformationcited by 3
5 more
  1. 2024Neural Precision Polarization: Simplifying Neural Network Inference with Dual-Level Precisioncited by 2
  2. 2025On the Duality between Gradient Transformations and Adapterscited by 1
  3. 2024MoIN: Mixture of Introvert Experts to Upcycle an LLMcited by 0
  4. 2025Towards Efficient Post-Training via Fourier-Driven Adapter Architecturescited by 0
  5. 2026Learned Subspace Compression for Communication-Efficient Pipeline Parallelismcited by 0