# Improving Quaternion Neural Networks with Quaternionic Activation Functions

arXiv: https://arxiv.org/abs/2406.16481 · HTML: https://mlsanity.com/paper/2406.16481/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 60 references that are in the map.
- First posted on arXiv 24 Jun 2024.

## Builds on (most cited first, 25 of 25)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)](https://mlsanity.com/paper/1511.07289/) (2015)
- [Activation Functions in Deep Learning: A Comprehensive Survey and Benchmark](https://mlsanity.com/paper/2109.14545/) (2021)
- [A survey on modern trainable activation functions](https://mlsanity.com/paper/2005.00817/) (2020)
- [A Survey of Complex-Valued Neural Networks](https://mlsanity.com/paper/2101.12249/) (2021)
- [Deep Quaternion Networks](https://mlsanity.com/paper/1712.04604/) (2017)
- [Quaternion Convolutional Neural Networks](https://mlsanity.com/paper/1903.00658/) (2019)
- [Quaternion Recurrent Neural Networks](https://mlsanity.com/paper/1806.04418/) (2018)
- [Better than Real: Complex-valued Neural Nets for MRI Fingerprinting](https://mlsanity.com/paper/1707.00070/) (2017)
- [Quaternion Convolutional Neural Networks for Heterogeneous Image Processing](https://mlsanity.com/paper/1811.02656/) (2018)
- [Lightweight and Efficient Neural Natural Language Processing with Quaternion Networks](https://mlsanity.com/paper/1906.04393/) (2019)
- [PHNNs: Lightweight Neural Networks via Parameterized Hypercomplex Convolutions](https://mlsanity.com/paper/2110.04176/) (2021)
- [Complex-valued Neural Networks with Non-parametric Activation Functions](https://mlsanity.com/paper/1802.08026/) (2018)
- [Quaternion Convolutional Neural Networks for Detection and Localization of 3D Sound Events](https://mlsanity.com/paper/1812.06811/) (2018)
- [A Quaternion-Valued Variational Autoencoder](https://mlsanity.com/paper/2010.11647/) (2020)
- [Speech Emotion Recognition Using Quaternion Convolutional Neural Networks](https://mlsanity.com/paper/2111.00404/) (2021)
- [Quaternion Gradient and Hessian](https://mlsanity.com/paper/1406.3587/) (2014)
- [Quaternion Collaborative Filtering for Recommendation](https://mlsanity.com/paper/1906.02594/) (2019)
- [Quaternion Factorization Machines: A Lightweight Solution to Intricate Feature Interaction Modelling](https://mlsanity.com/paper/2104.01716/) (2021)
- [Compressing deep quaternion neural networks with targeted regularization](https://mlsanity.com/paper/1907.11546/) (2019)
- [Quaternion-Valued Recurrent Projection Neural Networks on Unit Quaternions](https://mlsanity.com/paper/2001.11846/) (2020)
- [Time Series Compression using Quaternion Valued Neural Networks and Quaternion Backpropagation](https://mlsanity.com/paper/2403.11722/) (2024)
- [Quaternion Capsule Networks](https://mlsanity.com/paper/2007.04389/) (2020)

## Built on by (most cited first, 1 of 1)

- [Quaternion Approximation Networks for Enhanced Image Classification and Oriented Object Detection](https://mlsanity.com/paper/2509.05512/) (2025)

## Abstract (arXiv)

In this paper, we propose novel quaternion activation functions where we modify either the quaternion magnitude or the phase, as an alternative to the commonly used split activation functions. We define criteria that are relevant for quaternion activation functions, and subsequently we propose our novel activation functions based on this analysis. Instead of applying a known activation function like the ReLU or Tanh on the quaternion elements separately, these activation functions consider the quaternion properties and respect the quaternion space mathbbH. In particular, all quaternion components are utilized to calculate all output components, carrying out the benefit of the Hamilton product in e.g. the quaternion convolution to the activation functions. The proposed activation functions can be incorporated in arbitrary quaternion valued neural networks trained with gradient descent techniques. We further discuss the derivatives of the proposed activation functions where we observe beneficial properties for the activation functions affecting the phase. Specifically, they prove to be sensitive on basically the whole input range, thus improved gradient flow can be expected. We provide an elaborate experimental evaluation of our proposed quaternion activation functions including comparison with the split ReLU and split Tanh on two image classification tasks using the CIFAR-10 and SVHN dataset. There, especially the quaternion activation functions affecting the phase consistently prove to provide better performance.