# Fast Walsh-Hadamard Transform and Smooth-Thresholding Based Binary Layers in Deep Neural Networks

arXiv: https://arxiv.org/abs/2104.07085 · HTML: https://mlsanity.com/paper/2104.07085/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 57 references that are in the map.
- First posted on arXiv 14 Apr 2021.

## Builds on (most cited first, 30 of 31)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning](https://mlsanity.com/paper/1602.07261/) (2016)
- [SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size](https://mlsanity.com/paper/1602.07360/) (2016)
- [XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks](https://mlsanity.com/paper/1603.05279/) (2016)
- [CCNet: Criss-Cross Attention for Semantic Segmentation](https://mlsanity.com/paper/1811.11721/) (2018)
- [BiSeNet: Bilateral Segmentation Network for Real-time Semantic Segmentation](https://mlsanity.com/paper/1808.00897/) (2018)
- [Residual Attention Network for Image Classification](https://mlsanity.com/paper/1704.06904/) (2017)
- [Quantized Convolutional Neural Networks for Mobile Devices](https://mlsanity.com/paper/1512.06473/) (2015)
- [ReActNet: Towards Precise Binary Neural Network with Generalized Activation Functions](https://mlsanity.com/paper/2003.03488/) (2020)
- [Compressing Neural Networks with the Hashing Trick](https://mlsanity.com/paper/1504.04788/) (2015)
- [Fast-SCNN: Fast Semantic Segmentation Network](https://mlsanity.com/paper/1902.04502/) (2019)
- [Training Binary Neural Networks with Real-to-Binary Convolutions](https://mlsanity.com/paper/2003.11535/) (2020)
- [Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1](https://mlsanity.com/paper/1602.02830/) (2016)
- [BATS: Binary ArchitecTure Search](https://mlsanity.com/paper/2003.01711/) (2020)
- [GradiVeQ: Vector Quantization for Bandwidth-Efficient Gradient Aggregation in Distributed CNN Training](https://mlsanity.com/paper/1811.03617/) (2018)
- [Larq Compute Engine: Design, Benchmark, and Deploy State-of-the-Art Binarized Neural Networks](https://mlsanity.com/paper/2011.09398/) (2020)
- [Designing Adaptive Neural Networks for Energy-Constrained Image Classification](https://mlsanity.com/paper/1808.01550/) (2018)
- [S2-BNN: Bridging the Gap Between Self-Supervised Real and 1-bit Neural Networks via Guided Distribution Calibration](https://mlsanity.com/paper/2102.08946/) (2021)
- [Energy Efficient Hadamard Neural Networks](https://mlsanity.com/paper/1805.05421/) (2018)
- [Building Efficient Deep Neural Networks with Unitary Group Convolutions](https://mlsanity.com/paper/1811.07755/) (2018)
- [Hierarchical binary CNNs for landmark localization with limited resources](https://mlsanity.com/paper/1808.04803/) (2018)
- [A Self-Attentive Emotion Recognition Network](https://mlsanity.com/paper/1905.01972/) (2019)

## Built on by (most cited first, 14 of 14)

- [Block Walsh-Hadamard Transform Based Binary Layers in Deep Neural Networks](https://mlsanity.com/paper/2201.02711/) (2022)
- [A Hybrid Quantum-Classical Approach based on the Hadamard Transform for the Convolutional Layer](https://mlsanity.com/paper/2305.17510/) (2023)
- [LLaVA-FA: Learning Fourier Approximation for Compressing Large Multimodal Models](https://mlsanity.com/paper/2602.00135/) (2026)
- [Optimization Guarantees of Unfolded ISTA and ADMM Networks With Smooth Soft-Thresholding](https://mlsanity.com/paper/2309.06195/) (2023)
- [DCT Perceptron Layer: A Transform Domain Approach for Convolution Layer](https://mlsanity.com/paper/2211.08577/) (2022)
- [ADC/DAC-Free Analog Acceleration of Deep Neural Networks with Frequency Transformation](https://mlsanity.com/paper/2309.01771/) (2023)
- [Multichannel Orthogonal Transform-Based Perceptron Layers for Efficient ResNets](https://mlsanity.com/paper/2303.06797/) (2023)
- [HTMA-Net: Towards Multiplication-Avoiding Neural Networks via Hadamard Transform and In-Memory Computing](https://mlsanity.com/paper/2509.23103/) (2025)
- [Multipod Convolutional Network](https://mlsanity.com/paper/2210.00689/) (2022)
- [Containing Analog Data Deluge at Edge through Frequency-Domain Compression in Collaborative Compute-in-Memory Networks](https://mlsanity.com/paper/2309.11048/) (2023)
- [WTHaar-Net: a Hybrid Quantum-Classical Approach](https://mlsanity.com/paper/2603.02497/) (2026)
- [ShearFuse-UNet: Hadamard, DCT, and Shearlet Transform Fusion for Next-Day Wildfire Spread Prediction](https://mlsanity.com/paper/2606.14071/) (2026)
- [Structure of the Circular-Dyadic Convolution Error](https://mlsanity.com/paper/2607.15293/) (2026)
- [WHTMix: Efficient Stereo Depth Estimation via Walsh-Hadamard Token Mixing](https://mlsanity.com/paper/2607.25234/) (2026)

## Abstract (arXiv)

In this paper, we propose a novel layer based on fast Walsh-Hadamard transform (WHT) and smooth-thresholding to replace 1× 1 convolution layers in deep neural networks. In the WHT domain, we denoise the transform domain coefficients using the new smooth-thresholding non-linearity, a smoothed version of the well-known soft-thresholding operator. We also introduce a family of multiplication-free operators from the basic 2×2 Hadamard transform to implement 3× 3 depthwise separable convolution layers. Using these two types of layers, we replace the bottleneck layers in MobileNet-V2 to reduce the network's number of parameters with a slight loss in accuracy. For example, by replacing the final third bottleneck layers, we reduce the number of parameters from 2.270M to 540K. This reduces the accuracy from 95.21% to 92.98% on the CIFAR-10 dataset. Our approach significantly improves the speed of data processing. The fast Walsh-Hadamard transform has a computational complexity of O(mlog2 m). As a result, it is computationally more efficient than the 1×1 convolution layer. The fast Walsh-Hadamard layer processes a tensor in ℝ10×32×32×1024 about 2 times faster than 1×1 convolution layer on NVIDIA Jetson Nano computer board.