# Transfer Learning Between Different Architectures Via Weights Injection

arXiv: https://arxiv.org/abs/2101.02757 · HTML: https://mlsanity.com/paper/2101.02757/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 6 references that are in the map.
- First posted on arXiv 7 Jan 2021.

## Builds on (most cited first, 5 of 5)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Net2Net: Accelerating Learning via Knowledge Transfer](https://mlsanity.com/paper/1511.05641/) (2015)
- [FNA++: Fast Network Adaptation via Parameter Remapping and Architecture Search](https://mlsanity.com/paper/2006.12986/) (2020)

## Built on by (most cited first, 3 of 3)

- [2-speed network ensemble for efficient classification of incremental land-use/land-cover satellite image chips](https://mlsanity.com/paper/2203.08267/) (2022)
- [SPRKD: Effective Knowledge Distillation for Deep Neural Networks via Saddle Region Approximation](https://mlsanity.com/paper/2607.23346/) (2026)
- [CrossGMN: Graph Metanetworks for Cross-Architecture Weight-Space Transformations](https://mlsanity.com/paper/2610.01649/) (2026)

## Abstract (arXiv)

This work presents a naive algorithm for parameter transfer between different architectures with a computationally cheap injection technique (which does not require data). The primary objective is to speed up the training of neural networks from scratch. It was found in this study that transferring knowledge from any architecture was superior to Kaiming and Xavier for initialization. In conclusion, the method presented is found to converge faster, which makes it a drop-in replacement for classical methods. The method involves: 1) matching: the layers of the pre-trained model with the targeted model; 2) injection: the tensor is transformed into a desired shape. This work provides a comparison of similarity between the current SOTA architectures (ImageNet), by utilising TLI (Transfer Learning by Injection) score.