# Building Blocks for a Complex-Valued Transformer Architecture

arXiv: https://arxiv.org/abs/2306.09827 · HTML: https://mlsanity.com/paper/2306.09827/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 37 references that are in the map.
- First posted on arXiv 16 Jun 2023.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021)
- [Efficient Transformers: A Survey](https://mlsanity.com/paper/2009.06732/) (2020)
- [FDA: Fourier Domain Adaptation for Semantic Segmentation](https://mlsanity.com/paper/2004.05498/) (2020)
- [Conditional Positional Encodings for Vision Transformers](https://mlsanity.com/paper/2102.10882/) (2021)
- [A Survey of Transformers](https://mlsanity.com/paper/2106.04554/) (2021)
- [A Fourier-based Framework for Domain Generalization](https://mlsanity.com/paper/2105.11120/) (2021)
- [Deep Complex Networks](https://mlsanity.com/paper/1705.09792/) (2017)
- [Music Transformer](https://mlsanity.com/paper/1809.04281/) (2018)
- [DCCRN: Deep Complex Convolution Recurrent Network for Phase-Aware Speech Enhancement](https://mlsanity.com/paper/2008.00264/) (2020)
- [Phase-aware Speech Enhancement with Deep Complex U-Net](https://mlsanity.com/paper/1903.03107/) (2019)
- [A Survey of Complex-Valued Neural Networks](https://mlsanity.com/paper/2101.12249/) (2021)
- [Learning Features of Music from Scratch](https://mlsanity.com/paper/1611.09827/) (2016)
- [Encoding word order in complex embeddings](https://mlsanity.com/paper/1912.12333/) (2019)
- [PHASEN: A Phase-and-Harmonics-Aware Speech Enhancement Network](https://mlsanity.com/paper/1911.04697/) (2019)
- [An Introductory Survey on Attention Mechanisms in NLP Problems](https://mlsanity.com/paper/1811.05544/) (2018)
- [Complex Transformer: A Framework for Modeling Complex-Valued Sequence](https://mlsanity.com/paper/1910.10202/) (2019)
- [Better than Real: Complex-valued Neural Nets for MRI Fingerprinting](https://mlsanity.com/paper/1707.00070/) (2017)
- [On Complex Valued Convolutional Neural Networks](https://mlsanity.com/paper/1602.09046/) (2016)

## Built on by (most cited first, 16 of 16)

- [CVKAN: Complex-Valued Kolmogorov-Arnold Networks](https://mlsanity.com/paper/2502.02417/) (2025)
- [iFairy: the First 2-bit Complex LLM with All Parameters in ±1, ± i](https://mlsanity.com/paper/2508.05571/) (2025)
- [Expanding Expressivity in Transformer Models with M"obiusAttention](https://mlsanity.com/paper/2409.12175/) (2024)
- [A Survey of Deep Learning for Complex Speech Spectrograms](https://mlsanity.com/paper/2505.08694/) (2025)
- [Holographic Transformers for Complex-Valued Signal Processing: Integrating Phase Interference into Self-Attention](https://mlsanity.com/paper/2509.19331/) (2025)
- [Fairy2i: Training Complex LLMs from Real LLMs with All Parameters in ± 1, ± i](https://mlsanity.com/paper/2512.02901/) (2025)
- [Towards White-Box Deep Wireless Sensing](https://mlsanity.com/paper/2507.21799/) (2025)
- [CoPE: A Lightweight Complex Positional Encoding](https://mlsanity.com/paper/2508.18308/) (2025)
- [EuleroDec: A Complex-Valued RVQ-VAE for Efficient and Robust Audio Coding](https://mlsanity.com/paper/2601.17517/) (2026)
- [Fourier Transformers for Latent Crystallographic Diffusion and Generative Modeling](https://mlsanity.com/paper/2602.12045/) (2026)
- [Flexformer: Flexible Linear Transformer with Learnable Attention Kernel](https://mlsanity.com/paper/2606.27748/) (2026)
- [DeepCSHAP: Utilizing Shapley Values to Explain Deep Complex-Valued Neural Networks](https://mlsanity.com/paper/2403.08428/) (2024)
- [Unfolding Framework with Complex-Valued Deformable Attention for High-Quality Computer-Generated Hologram Generation](https://mlsanity.com/paper/2508.21657/) (2025)
- [Perturbing the Phase: Analyzing Adversarial Robustness of Complex-Valued Neural Networks](https://mlsanity.com/paper/2602.06577/) (2026)
- [Exploring Polarimetric Properties Preservation during Reconstruction of PolSAR images using Complex-valued Convolutional Neural Networks](https://mlsanity.com/paper/2602.07094/) (2026)
- [IQ-JEPA: A Joint-Embedding Predictive Architecture with a Hermitian Vision Transformer for Sound Speed and Attenuation Estimation from Ultrasound IQ Data](https://mlsanity.com/paper/2607.22351/) (2026)

## Abstract (arXiv)

Most deep learning pipelines are built on real-valued operations to deal with real-valued inputs such as images, speech or music signals. However, a lot of applications naturally make use of complex-valued signals or images, such as MRI or remote sensing. Additionally the Fourier transform of signals is complex-valued and has numerous applications. We aim to make deep learning directly applicable to these complex-valued signals without using projections into ℝ2. Thus we add to the recent developments of complex-valued neural networks by presenting building blocks to transfer the transformer architecture to the complex domain. We present multiple versions of a complex-valued Scaled Dot-Product Attention mechanism as well as a complex-valued layer normalization. We test on a classification and a sequence generation task on the MusicNet dataset and show improved robustness to overfitting while maintaining on-par performance when compared to the real-valued transformer architecture.