mlsanityData to 13 Oct 2026

Initializing Models with Larger Ones

arXiv 2311.18823 ↗First posted 30 Nov 2023cs.LG cs.CVcited by 39

Not on the map. Why?

Abstract (arXiv)

Weight initialization plays an important role in neural network training. Widely used initialization methods are proposed and evaluated for networks that are trained from scratch. However, the growing number of pretrained models now offers new opportunities for tackling this classical problem of weight initialization. In this work, we introduce weight selection, a method for initializing smaller models by selecting a subset of weights from a pretrained larger model. This enables the transfer of knowledge from pretrained weights to smaller models. Our experiments demonstrate that weight selection can significantly enhance the performance of small models and reduce their training time. Notably, it can also be used together with knowledge distillation. Weight selection offers a new approach to leverage the power of pretrained models in resource-constrained settings, and we hope it can be a useful tool for training small models in the large-model era. Code is available at https://github.com/OscarXZQ/weight-selection.

Key facts. Cited by 39 papers in the map (data through 13 Oct 2026). Builds on 54 of its 69 references that are in the map. First posted on arXiv 30 Nov 2023.

Builds on

54 of 69 references.

References · 54

  1. 2017Attention Is All You Needcited by 60,106
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,950
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,588
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,043
  6. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,223
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,072
  8. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,111
  9. 2015Distilling the Knowledge in a Neural Networkcited by 10,180
  10. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,364
  11. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,915
  12. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,457
38 more
  1. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,170
  2. 2020Training data-efficient image transformers & distillation through attentioncited by 4,574
  3. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,281
  4. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,141
  5. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,601
  6. 2022A ConvNet for the 2020scited by 3,563
  7. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,882
  8. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,544
  9. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,368
  10. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,266
  11. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,215
  12. 2013Describing Textures in the Wildcited by 2,115
  13. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,017
  14. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,492
  15. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,442
  16. 2017Random Erasing Data Augmentationcited by 1,238
  17. 2019A Comprehensive Survey on Transfer Learningcited by 1,157
  18. 2016Pruning Filters for Efficient ConvNetscited by 1,138
  19. 2014Two-Stream Convolutional Networks for Action Recognition in Videoscited by 1,067
  20. 2016Deep Networks with Stochastic Depthcited by 977
  21. 2019Big Transfer (BiT): General Visual Representation Learningcited by 768
  22. 2021Going deeper with Image Transformerscited by 762
  23. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 755
  24. 2019Contrastive Representation Distillationcited by 739
  25. 2018Rethinking the Value of Network Pruningcited by 597
  26. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 566
  27. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 409
  28. 2021Knowledge distillation: A good teacher is patient and consistentcited by 255
  29. 2015Net2Net: Accelerating Learning via Knowledge Transfercited by 246
  30. 2015All you need is a good initcited by 124
  31. 2016Fully Convolutional Instance-aware Semantic Segmentationcited by 122
  32. 2017On orthogonality and learning recurrent networks with long term dependenciescited by 82
  33. 2020Pre-trained Summarization Distillationcited by 70
  34. 2021bert2BERT: Towards Reusable Pretrained Language Modelscited by 70
  35. 2023Mimetic Initialization of Self-Attention Layerscited by 53
  36. 2019Selfie: Self-supervised Pretraining for Image Embeddingcited by 46
  37. 2022How to Train Vision Transformer on Small-scale Datasets?cited by 34
  38. 2015Data-dependent Initializations of Convolutional Neural Networkscited by 29

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 11; 2025: 13; 2026: 15.

39 papers.

Sort:

By area:

  1. 2024Efficient Learning With Sine-Activated Low-rank Matricescited by 22
  2. 2024WAVE: Weight Templates for Adaptive Initialization of Variable-sized Modelscited by 16
  3. 2025MergeVQ: A Unified Framework for Visual Generation and Representation with Disentangled Token Merging and Quantizationcited by 12
  4. 2024Accessing Vision Foundation Models via ImageNet-1Kcited by 10
  5. 2026Procedural Pretraining: Warming Up Language Models with Abstract Datacited by 10
  6. 2024Exploring Learngene via Stage-wise Weight Sharing for Initializing Variable-sized Modelscited by 9
  7. 2024MoE Jetpack: From Dense Checkpoints to Adaptive Mixture of Experts for Vision Taskscited by 9
  8. 2025Transformers Pretrained on Procedural Data Contain Modular Structures for Algorithmic Reasoningcited by 9
  9. 2025Cutting the Skip: Training Residual-Free Transformerscited by 8
  10. 2025Can You Learn to See Without Images? Procedural Warm-Up for Vision Transformerscited by 8
  11. 2025Towards Superior Quantization Accuracy: A Layer-sensitive Approachcited by 7
  12. 2025Structured Initialization for Vision Transformerscited by 6
27 more
  1. 2024FINE: Factorizing Knowledge for Initialization of Variable-sized Diffusion Modelscited by 5
  2. 2025One-Shot Knowledge Transfer for Scalable Person Re-Identificationcited by 4
  3. 2024Adapting LLaMA Decoder to Vision Transformercited by 3
  4. 2024JPEG Inspired Deep Learningcited by 3
  5. 2024FedBaF: Federated Learning Aggregation Biased by a Foundation Modelcited by 3
  6. 2024The Master Key Filters Hypothesis: Deep Filters Are Generalcited by 3
  7. 2025A Model Zoo of Vision Transformerscited by 3
  8. 2026Attention Transfer Is Not Universally Effective for Vision Transformerscited by 3
  9. 2024Discrete Cosine Transform Based Decorrelated Attention for Vision Transformerscited by 2
  10. 2025CoRe^2: Collect, Reflect and Refine to Generate Better and Fastercited by 2
  11. 2025GUIDE: Guided Initialization and Distillation of Embeddingscited by 1
  12. 2026A Unified Framework for Knowledge Transfer in Bidirectional Model Scalingcited by 1
  13. 2026Making Transformers Logicalcited by 1
  14. 2025Projectable Models: One-Shot Generation of Small Specialized Transformers from Large Onescited by 0
  15. 2025Dual-Model Weight Selection and Self-Knowledge Distillation for Medical Image Classificationcited by 0
  16. 2025Delta Activations: A Representation for Finetuned Large Language Modelscited by 0
  17. 2026Self-Supervised Weight Templates for Scalable Vision Model Initializationcited by 0
  18. 2026A Unified Revisit of Temperature in Classification-Based Knowledge Distillationcited by 0
  19. 2026Breaking the Scale Barrier: One-Shot Knowledge Transfer via Frequency Transformcited by 0
  20. 2026Efficient Unlearning through Maximizing Relearning Convergence Delaycited by 0
  21. 2026Chain-of-Models Pre-Training: Rethinking Training Acceleration of Vision Foundation Modelscited by 0
  22. 2026Constraint-based Pre-training: From Structured Constraints to Scalable Model Initializationcited by 0
  23. 2026Small LLMs: Pruning vs. Training from Scratchcited by 0
  24. 2026Conditioned Initialization for Attentioncited by 0
  25. 2026Copy the Same, Distill the Difference: Initializing Linear Vision Transformerscited by 0
  26. 2026Procedural Core: A Compact Recurrent Initialization for Vision Transformerscited by 0
  27. 2026CrossGMN: Graph Metanetworks for Cross-Architecture Weight-Space Transformationscited by 0