# Non-Iterative Knowledge Fusion in Deep Convolutional Neural Networks

arXiv: https://arxiv.org/abs/1809.09399 · HTML: https://mlsanity.com/paper/1809.09399/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Sep 2018.

## Built on by (most cited first, 7 of 7)

- [Deep Model Fusion: A Survey](https://mlsanity.com/paper/2309.15698/) (2023)
- [On Cross-Layer Alignment for Model Fusion of Heterogeneous Neural Networks](https://mlsanity.com/paper/2110.15538/) (2021)
- [Continual Learning of Generative Models with Limited Data: From Wasserstein-1 Barycenter to Adaptive Coalescence](https://mlsanity.com/paper/2101.09225/) (2021)
- [Test-time Collective Prediction](https://mlsanity.com/paper/2106.12012/) (2021)
- [Multi-Task Structural Learning using Local Task Similarity induced Neuron Creation and Removal](https://mlsanity.com/paper/2305.00441/) (2023)
- [Toward Data Efficient Model Merging between Different Datasets without Performance Degradation](https://mlsanity.com/paper/2306.05641/) (2023)
- [AKE-GNN: Effective Graph Learning with Adaptive Knowledge Exchange](https://mlsanity.com/paper/2106.05455/) (2021)

## Abstract (arXiv)

Incorporation of a new knowledge into neural networks with simultaneous preservation of the previous one is known to be a nontrivial problem. This problem becomes even more complex when new knowledge is contained not in new training examples, but inside the parameters (connection weights) of another neural network. Here we propose and test two methods allowing combining the knowledge contained in separate networks. One method is based on a simple operation of summation of weights of constituent neural networks. Another method assumes incorporation of a new knowledge by modification of weights nonessential for the preservation of already stored information. We show that with these methods the knowledge from one network can be transferred into another one non-iteratively without requiring training sessions. The fused network operates efficiently, performing classification far better than a chance level. The efficiency of the methods is quantified on several publicly available data sets in classification tasks both for shallow and deep neural networks.