# PHYDI: Initializing Parameterized Hypercomplex Neural Networks as Identity Functions

arXiv: https://arxiv.org/abs/2310.07612 · HTML: https://mlsanity.com/paper/2310.07612/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 20 references that are in the map.
- First posted on arXiv 11 Oct 2023.

## Builds on (most cited first, 13 of 13)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [On Layer Normalization in the Transformer Architecture](https://mlsanity.com/paper/2002.04745/) (2020)
- [ReZero is All You Need: Fast Convergence at Large Depth](https://mlsanity.com/paper/2003.04887/) (2020)
- [Deep Complex Networks](https://mlsanity.com/paper/1705.09792/) (2017)
- [Fixup Initialization: Residual Learning Without Normalization](https://mlsanity.com/paper/1901.09321/) (2019)
- [Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice](https://mlsanity.com/paper/1711.04735/) (2017)
- [Beyond Fully-Connected Layers with Quaternions: Parameterization of Hypercomplex Multiplications with 1/n Parameters](https://mlsanity.com/paper/2102.08597/) (2021)
- [PHNNs: Lightweight Neural Networks via Parameterized Hypercomplex Convolutions](https://mlsanity.com/paper/2110.04176/) (2021)
- [Dual Quaternion Ambisonics Array for Six-Degree-of-Freedom Acoustic Representation](https://mlsanity.com/paper/2204.01851/) (2022)
- [Multi-View Hypercomplex Learning for Breast Cancer Screening](https://mlsanity.com/paper/2204.05798/) (2022)
- [Predicting Rigid Body Dynamics using Dual Quaternion Recurrent Neural Networks with Quaternion Attention](https://mlsanity.com/paper/2011.08734/) (2020)
- [Hypercomplex Image-to-Image Translation](https://mlsanity.com/paper/2205.02087/) (2022)

## Built on by (most cited first, 2 of 2)

- [Towards Explaining Hypercomplex Neural Networks](https://mlsanity.com/paper/2403.17929/) (2024)
- [PHemoNet: A Multimodal Network for Physiological Signals](https://mlsanity.com/paper/2410.00010/) (2024)

## Abstract (arXiv)

Neural models based on hypercomplex algebra systems are growing and prolificating for a plethora of applications, ranging from computer vision to natural language processing. Hand in hand with their adoption, parameterized hypercomplex neural networks (PHNNs) are growing in size and no techniques have been adopted so far to control their convergence at a large scale. In this paper, we study PHNNs convergence and propose parameterized hypercomplex identity initialization (PHYDI), a method to improve their convergence at different scales, leading to more robust performance when the number of layers scales up, while also reaching the same performance with fewer iterations. We show the effectiveness of this approach in different benchmarks and with common PHNNs with ResNets- and Transformer-based architecture. The code is available at https://github.com/ispamm/PHYDI.