# A Unified Weight Initialization Paradigm for Tensorial Convolutional Neural Networks

arXiv: https://arxiv.org/abs/2205.15307 · HTML: https://mlsanity.com/paper/2205.15307/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 32 references that are in the map.
- First posted on arXiv 28 May 2022.

## Builds on (most cited first, 24 of 24)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [MLP-Mixer: An all-MLP Architecture for Vision](https://mlsanity.com/paper/2105.01601/) (2021)
- [Striving for Simplicity: The All Convolutional Net](https://mlsanity.com/paper/1412.6806/) (2014)
- [Pay Attention to MLPs](https://mlsanity.com/paper/2105.08050/) (2021)
- [Tensorizing Neural Networks](https://mlsanity.com/paper/1509.06569/) (2015)
- [Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decomposition](https://mlsanity.com/paper/1412.6553/) (2014)
- [Compression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applications](https://mlsanity.com/paper/1511.06530/) (2015)
- [Ultimate tensorization: compressing convolutional and FC layers alike](https://mlsanity.com/paper/1611.03214/) (2016)
- [Towards Efficient Tensor Decomposition-Based DNN Model Compression with Optimization Framework](https://mlsanity.com/paper/2107.12422/) (2021)
- [Wide Compression: Tensor Ring Nets](https://mlsanity.com/paper/1802.09052/) (2018)
- [Factorized Higher-Order CNNs with an Application to Spatio-Temporal Emotion Estimation](https://mlsanity.com/paper/1906.06196/) (2019)
- [Compressing Recurrent Neural Networks with Tensor Ring for Action Recognition](https://mlsanity.com/paper/1811.07503/) (2018)
- [Learning Compact Recurrent Neural Networks with Block-Term Tensor Decomposition](https://mlsanity.com/paper/1712.05134/) (2017)
- [Compressing deep neural networks by matrix product operators](https://mlsanity.com/paper/1904.06194/) (2019)
- [Heuristic Rank Selection with Progressively Searching Tensor Ring Network](https://mlsanity.com/paper/2009.10580/) (2020)
- [Einconv: Exploring Unexplored Tensor Network Decompositions for Convolutional Neural Networks](https://mlsanity.com/paper/1908.04471/) (2019)
- [Block-term Tensor Neural Networks](https://mlsanity.com/paper/2010.04963/) (2020)
- [Deep Residual Networks and Weight Initialization](https://mlsanity.com/paper/1709.02956/) (2017)
- [TedNet: A Pytorch Toolkit for Tensor Decomposition Networks](https://mlsanity.com/paper/2104.05018/) (2021)
- [Accelerating Training using Tensor Decomposition](https://mlsanity.com/paper/1909.05675/) (2019)

## Built on by (most cited first, 8 of 8)

- [Tensor Networks Meet Neural Networks: A Survey and Future Perspectives](https://mlsanity.com/paper/2302.09019/) (2023)
- [Compute Better Spent: Replacing Dense Layers with Structured Matrices](https://mlsanity.com/paper/2406.06248/) (2024)
- [Reusing Pretrained Models by Multi-linear Operators for Efficient Training](https://mlsanity.com/paper/2310.10699/) (2023)
- [Low Rank Optimization for Efficient Deep Learning: Making A Balance between Compact Architecture and Fast Training](https://mlsanity.com/paper/2303.13635/) (2023)
- [IDInit: A Universal and Stable Initialization Method for Neural Network Training](https://mlsanity.com/paper/2503.04626/) (2025)
- [Expression Syntax Information Bottleneck for Math Word Problems](https://mlsanity.com/paper/2310.15664/) (2023)
- [Advocating for the Silent: Enhancing Federated Generalization for Non-Participating Clients](https://mlsanity.com/paper/2310.07171/) (2023)
- [An Effective Weight Initialization Method for Deep Learning: Application to Satellite Image Classification](https://mlsanity.com/paper/2406.00348/) (2024)

## Abstract (arXiv)

Tensorial Convolutional Neural Networks (TCNNs) have attracted much research attention for their power in reducing model parameters or enhancing the generalization ability. However, exploration of TCNNs is hindered even from weight initialization methods. To be specific, general initialization methods, such as Xavier or Kaiming initialization, usually fail to generate appropriate weights for TCNNs. Meanwhile, although there are ad-hoc approaches for specific architectures (e.g., Tensor Ring Nets), they are not applicable to TCNNs with other tensor decomposition methods (e.g., CP or Tucker decomposition). To address this problem, we propose a universal weight initialization paradigm, which generalizes Xavier and Kaiming methods and can be widely applicable to arbitrary TCNNs. Specifically, we first present the Reproducing Transformation to convert the backward process in TCNNs to an equivalent convolution process. Then, based on the convolution operators in the forward and backward processes, we build a unified paradigm to control the variance of features and gradients in TCNNs. Thus, we can derive fan-in and fan-out initialization for various TCNNs. We demonstrate that our paradigm can stabilize the training of TCNNs, leading to faster convergence and better results.