mlsanityData to 14 Oct 2026

Reusing Pretrained Models by Multi-linear Operators for Efficient Training

arXiv 2310.10699 ↗First posted 16 Oct 2023cs.LG cs.AIcited by 15

Not on the map. Why?

Abstract (arXiv)

Training large models from scratch usually costs a substantial amount of resources. Towards this problem, recent studies such as bert2BERT and LiGO have reused small pretrained models to initialize a large model (termed the ``target model''), leading to a considerable acceleration in training. Despite the successes of these previous studies, they grew pretrained models by mapping partial weights only, ignoring potential correlations across the entire model. As we show in this paper, there are inter- and intra-interactions among the weights of both the pretrained and the target models. As a result, the partial mapping may not capture the complete information and lead to inadequate growth. In this paper, we propose a method that linearly correlates each weight of the target model to all the weights of the pretrained model to further enhance acceleration ability. We utilize multi-linear operators to reduce computational and spacial complexity, enabling acceptable resource requirements. Experiments demonstrate that our method can save 76% computational costs on DeiT-base transferred from DeiT-small, which outperforms bert2BERT by +12.0% and LiGO by +20.7%, respectively.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Builds on 52 of its 72 references that are in the map. First posted on arXiv 16 Oct 2023.

Builds on

52 of 72 references.

References · 52

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  7. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  8. 2020Scaling Laws for Neural Language Modelscited by 6,586
  9. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  10. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  11. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  12. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
38 more
  1. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  2. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
  3. 2021Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentationcited by 1,222
  4. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,043
  5. 2022Token Merging: Your ViT But Fastercited by 1,021
  6. 2017ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseasescited by 972
  7. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 674
  8. 2022Scaling Vision Transformers to Gigapixel Images via Hierarchical Self-Supervised Learningcited by 359
  9. 2020ReZero is All You Need: Fast Convergence at Large Depthcited by 264
  10. 2015Net2Net: Accelerating Learning via Knowledge Transfercited by 250
  11. 2019A Constructive Prediction of the Generalization Error Across Scalescited by 210
  12. 2019Fixup Initialization: Residual Learning Without Normalizationcited by 198
  13. 2021TransGAN: Two Pure Transformers Can Make One Strong GAN, and That Can Scale Upcited by 162
  14. 2021FedPara: Low-Rank Hadamard Product for Communication-Efficient Federated Learningcited by 141
  15. 2016Identity Matters in Deep Learningcited by 109
  16. 2020Accelerating Training of Transformer-Based Language Models with Progressive Layer Droppingcited by 93
  17. 2019A Tensorized Transformer for Language Modelingcited by 86
  18. 2023Learning to Grow Pretrained Models for Efficient Transformer Trainingcited by 83
  19. 2022When Federated Learning Meets Pre-trained Language Models' Parameter-Efficient Tuning Methodscited by 75
  20. 2021bert2BERT: Towards Reusable Pretrained Language Modelscited by 70
  21. 2018SuperNeurons: Dynamic GPU Memory Management for Training Deep Neural Networkscited by 55
  22. 2019Splitting Steepest Descent for Growing Neural Architecturescited by 55
  23. 2021Towards Efficient Tensor Decomposition-Based DNN Model Compression with Optimization Frameworkcited by 52
  24. 2020On the Transformer Growth for Progressive BERT Trainingcited by 49
  25. 2021Firefly Neural Architecture Descent: a General Approach for Growing Neural Networkscited by 49
  26. 2022Staged Training for Transformer Language Modelscited by 49
  27. 2022EfficientTrain: Exploring Generalized Curriculum Learning for Training Visual Backbonescited by 43
  28. 2021Knowledge Inheritance for Pre-trained Language Modelscited by 38
  29. 2022Automated Progressive Learning for Efficient Training of Vision Transformerscited by 37
  30. 2018Compressing Recurrent Neural Networks with Tensor Ring for Action Recognitioncited by 34
  31. 2023Tensor Networks Meet Neural Networks: A Survey and Future Perspectivescited by 32
  32. 2020Progressively Stacking 2.0: A Multi-stage Layerwise Training Method for BERT Training Speedupcited by 31
  33. 2022Exploring Low Rank Training of Deep Neural Networkscited by 30
  34. 2020Heuristic Rank Selection with Progressively Searching Tensor Ring Networkcited by 28
  35. 2019T-Net: Parametrizing Fully Convolutional Nets with a Single High-Order Tensorcited by 22
  36. 2019Energy-Aware Neural Architecture Optimization with Fast Splitting Steepest Descentcited by 18
  37. 2020Steepest Descent Neural Architecture Optimization: Escaping Local Optimum with Signed Neural Splittingcited by 14
  38. 2020Block-term Tensor Neural Networkscited by 13

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 7; 2025: 3; 2026: 3.

15 papers.

Sort:

By area:

  1. 2023Efficient Large Language Models: A Surveycited by 186
  2. 2023SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scalingcited by 172
  3. 2024Upcycling Large Language Models into Mixture of Expertscited by 49
  4. 2024Measuring Vision-Language STEM Skills of Neural Modelscited by 14
  5. 2025Do we really have to filter out random noise in pre-training data for language models?cited by 11
  6. 2024Preparing Lessons for Progressive Training on Language Modelscited by 10
  7. 2024Beyond Next Token Prediction: Patch-Level Training for Large Language Modelscited by 7
  8. 2025Benchmarking Ultra-Low-Power μNPUscited by 6
  9. 2024Efficient Training of Large Vision Models via Advanced Automated Progressive Learningcited by 5
  10. 2025IDInit: A Universal and Stable Initialization Method for Neural Network Trainingcited by 5
  11. 2026SPARKLING: Balancing Signal Preservation and Symmetry Breaking for Width-Progressive Learningcited by 4
  12. 2024A framework for measuring the training efficiency of a neural architecturecited by 2
3 more
  1. 2026A Unified Framework for Knowledge Transfer in Bidirectional Model Scalingcited by 2
  2. 2024Long Sequence Modeling with Attention Tensorization: From Sequence to Tensor Learningcited by 1
  3. 2026Expert Upcycling: Shifting the Compute-Efficient Frontier of Mixture-of-Expertscited by 1