# Block Walsh-Hadamard Transform Based Binary Layers in Deep Neural Networks

arXiv: https://arxiv.org/abs/2201.02711 · HTML: https://mlsanity.com/paper/2201.02711/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 62 references that are in the map.
- First posted on arXiv 7 Jan 2022.

## Builds on (most cited first, 30 of 39)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [Searching for MobileNetV3](https://mlsanity.com/paper/1905.02244/) (2019)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning](https://mlsanity.com/paper/1602.07261/) (2016)
- [SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size](https://mlsanity.com/paper/1602.07360/) (2016)
- [XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks](https://mlsanity.com/paper/1603.05279/) (2016)
- [CCNet: Criss-Cross Attention for Semantic Segmentation](https://mlsanity.com/paper/1811.11721/) (2018)
- [BiSeNet: Bilateral Segmentation Network for Real-time Semantic Segmentation](https://mlsanity.com/paper/1808.00897/) (2018)
- [Residual Attention Network for Image Classification](https://mlsanity.com/paper/1704.06904/) (2017)
- [FNet: Mixing Tokens with Fourier Transforms](https://mlsanity.com/paper/2105.03824/) (2021)
- [Quantized Convolutional Neural Networks for Mobile Devices](https://mlsanity.com/paper/1512.06473/) (2015)
- [ReActNet: Towards Precise Binary Neural Network with Generalized Activation Functions](https://mlsanity.com/paper/2003.03488/) (2020)
- [Compressing Neural Networks with the Hashing Trick](https://mlsanity.com/paper/1504.04788/) (2015)
- [Towards Accurate Binary Convolutional Neural Network](https://mlsanity.com/paper/1711.11294/) (2017)
- [Fast-SCNN: Fast Semantic Segmentation Network](https://mlsanity.com/paper/1902.04502/) (2019)
- [Training Binary Neural Networks with Real-to-Binary Convolutions](https://mlsanity.com/paper/2003.11535/) (2020)
- [Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1](https://mlsanity.com/paper/1602.02830/) (2016)
- [Local Binary Convolutional Neural Networks](https://mlsanity.com/paper/1608.06049/) (2016)
- [BATS: Binary ArchitecTure Search](https://mlsanity.com/paper/2003.01711/) (2020)
- [GradiVeQ: Vector Quantization for Bandwidth-Efficient Gradient Aggregation in Distributed CNN Training](https://mlsanity.com/paper/1811.03617/) (2018)
- [Larq Compute Engine: Design, Benchmark, and Deploy State-of-the-Art Binarized Neural Networks](https://mlsanity.com/paper/2011.09398/) (2020)
- [Designing Adaptive Neural Networks for Energy-Constrained Image Classification](https://mlsanity.com/paper/1808.01550/) (2018)

## Built on by (most cited first, 13 of 13)

- [A Hybrid Quantum-Classical Approach based on the Hadamard Transform for the Convolutional Layer](https://mlsanity.com/paper/2305.17510/) (2023)
- [Fast-FNet: Accelerating Transformer Encoder Models via Efficient Fourier Layers](https://mlsanity.com/paper/2209.12816/) (2022)
- [DCT Perceptron Layer: A Transform Domain Approach for Convolution Layer](https://mlsanity.com/paper/2211.08577/) (2022)
- [ADC/DAC-Free Analog Acceleration of Deep Neural Networks with Frequency Transformation](https://mlsanity.com/paper/2309.01771/) (2023)
- [Multichannel Orthogonal Transform-Based Perceptron Layers for Efficient ResNets](https://mlsanity.com/paper/2303.06797/) (2023)
- [HTMA-Net: Towards Multiplication-Avoiding Neural Networks via Hadamard Transform and In-Memory Computing](https://mlsanity.com/paper/2509.23103/) (2025)
- [Multipod Convolutional Network](https://mlsanity.com/paper/2210.00689/) (2022)
- [Containing Analog Data Deluge at Edge through Frequency-Domain Compression in Collaborative Compute-in-Memory Networks](https://mlsanity.com/paper/2309.11048/) (2023)
- [NdLinear: Preserving Multi-Dimensional Structure for Parameter-Efficient Neural Networks](https://mlsanity.com/paper/2503.17353/) (2025)
- [Toward Generalizable Deep Learning Based Peatland Fire Detection via Walsh Hadamard Transform and Domain Adaptation](https://mlsanity.com/paper/2603.02465/) (2026)
- [WTHaar-Net: a Hybrid Quantum-Classical Approach](https://mlsanity.com/paper/2603.02497/) (2026)
- [A Large Scale Open-Source Image and Video Dataset for Robust Wildfire Detection and Classification](https://mlsanity.com/paper/2606.10174/) (2026)
- [Structure of the Circular-Dyadic Convolution Error](https://mlsanity.com/paper/2607.15293/) (2026)

## Abstract (arXiv)

Convolution has been the core operation of modern deep neural networks. It is well-known that convolutions can be implemented in the Fourier Transform domain. In this paper, we propose to use binary block Walsh-Hadamard transform (WHT) instead of the Fourier transform. We use WHT-based binary layers to replace some of the regular convolution layers in deep neural networks. We utilize both one-dimensional (1-D) and two-dimensional (2-D) binary WHTs in this paper. In both 1-D and 2-D layers, we compute the binary WHT of the input feature map and denoise the WHT domain coefficients using a nonlinearity which is obtained by combining soft-thresholding with the tanh function. After denoising, we compute the inverse WHT. We use 1D-WHT to replace the 1× 1 convolutional layers, and 2D-WHT layers can replace the 3×3 convolution layers and Squeeze-and-Excite layers. 2D-WHT layers with trainable weights can be also inserted before the Global Average Pooling (GAP) layers to assist the dense layers. In this way, we can reduce the number of trainable parameters significantly with a slight decrease in trainable parameters. In this paper, we implement the WHT layers into MobileNet-V2, MobileNet-V3-Large, and ResNet to reduce the number of parameters significantly with negligible accuracy loss. Moreover, according to our speed test, the 2D-FWHT layer runs about 24 times as fast as the regular 3× 3 convolution with 19.51% less RAM usage in an NVIDIA Jetson Nano experiment.