mlsanityData to 7 Oct 2026

Fast Walsh-Hadamard Transform and Smooth-Thresholding Based Binary Layers in Deep Neural Networks

arXiv 2104.07085 ↗First posted 14 Apr 2021cs.CV eess.IVcited by 14

Not on the map. Why?

Abstract (arXiv)

In this paper, we propose a novel layer based on fast Walsh-Hadamard transform (WHT) and smooth-thresholding to replace 1× 1 convolution layers in deep neural networks. In the WHT domain, we denoise the transform domain coefficients using the new smooth-thresholding non-linearity, a smoothed version of the well-known soft-thresholding operator. We also introduce a family of multiplication-free operators from the basic 2×2 Hadamard transform to implement 3× 3 depthwise separable convolution layers. Using these two types of layers, we replace the bottleneck layers in MobileNet-V2 to reduce the network's number of parameters with a slight loss in accuracy. For example, by replacing the final third bottleneck layers, we reduce the number of parameters from 2.270M to 540K. This reduces the accuracy from 95.21% to 92.98% on the CIFAR-10 dataset. Our approach significantly improves the speed of data processing. The fast Walsh-Hadamard transform has a computational complexity of O(mlog2 m). As a result, it is computationally more efficient than the 1×1 convolution layer. The fast Walsh-Hadamard layer processes a tensor in ℝ10×32×32×1024 about 2 times faster than 1×1 convolution layer on NVIDIA Jetson Nano computer board.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 31 of its 57 references that are in the map. First posted on arXiv 14 Apr 2021.

Builds on

31 of 57 references.

References · 31

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  4. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,829
  5. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  6. 2014Going Deeper with Convolutionscited by 4,231
  7. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  8. 2016Xception: Deep Learning with Depthwise Separable Convolutionscited by 2,320
  9. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  10. 2016Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learningcited by 1,509
  11. 2016SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model sizecited by 1,054
  12. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 829
19 more
  1. 2018CCNet: Criss-Cross Attention for Semantic Segmentationcited by 717
  2. 2018BiSeNet: Bilateral Segmentation Network for Real-time Semantic Segmentationcited by 645
  3. 2017Residual Attention Network for Image Classificationcited by 385
  4. 2015Quantized Convolutional Neural Networks for Mobile Devicescited by 212
  5. 2020ReActNet: Towards Precise Binary Neural Network with Generalized Activation Functionscited by 183
  6. 2015Compressing Neural Networks with the Hashing Trickcited by 150
  7. 2019Fast-SCNN: Fast Semantic Segmentation Networkcited by 124
  8. 2020Training Binary Neural Networks with Real-to-Binary Convolutionscited by 111
  9. 2016Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1cited by 102
  10. 2020BATS: Binary ArchitecTure Searchcited by 26
  11. 2018GradiVeQ: Vector Quantization for Bandwidth-Efficient Gradient Aggregation in Distributed CNN Trainingcited by 21
  12. 2020Larq Compute Engine: Design, Benchmark, and Deploy State-of-the-Art Binarized Neural Networkscited by 20
  13. 2018Designing Adaptive Neural Networks for Energy-Constrained Image Classificationcited by 18
  14. 2021S2-BNN: Bridging the Gap Between Self-Supervised Real and 1-bit Neural Networks via Guided Distribution Calibrationcited by 15
  15. 2018Energy Efficient Hadamard Neural Networkscited by 12
  16. 2018Building Efficient Deep Neural Networks with Unitary Group Convolutionscited by 9
  17. 2018Hierarchical binary CNNs for landmark localization with limited resourcescited by 7
  18. 2019A Self-Attentive Emotion Recognition Networkcited by 5
  19. 2019Robust and Computationally-Efficient Anomaly Detection using Powers-of-Two Networkscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 5; 2024: 0; 2025: 1; 2026: 5.

14 papers.

Sort:

By area:

  1. 2022Block Walsh-Hadamard Transform Based Binary Layers in Deep Neural Networkscited by 13
  2. 2023A Hybrid Quantum-Classical Approach based on the Hadamard Transform for the Convolutional Layercited by 12
  3. 2026LLaVA-FA: Learning Fourier Approximation for Compressing Large Multimodal Modelscited by 10
  4. 2022DCT Perceptron Layer: A Transform Domain Approach for Convolution Layercited by 3
  5. 2023ADC/DAC-Free Analog Acceleration of Deep Neural Networks with Frequency Transformationcited by 3
  6. 2023Optimization Guarantees of Unfolded ISTA and ADMM Networks With Smooth Soft-Thresholdingcited by 3
  7. 2023Multichannel Orthogonal Transform-Based Perceptron Layers for Efficient ResNetscited by 2
  8. 2025HTMA-Net: Towards Multiplication-Avoiding Neural Networks via Hadamard Transform and In-Memory Computingcited by 1
  9. 2022Multipod Convolutional Networkcited by 0
  10. 2023Containing Analog Data Deluge at Edge through Frequency-Domain Compression in Collaborative Compute-in-Memory Networkscited by 0
  11. 2026WTHaar-Net: a Hybrid Quantum-Classical Approachcited by 0
  12. 2026ShearFuse-UNet: Hadamard, DCT, and Shearlet Transform Fusion for Next-Day Wildfire Spread Predictioncited by 0
2 more
  1. 2026Structure of the Circular-Dyadic Convolution Errorcited by 0
  2. 2026WHTMix: Efficient Stereo Depth Estimation via Walsh-Hadamard Token Mixingcited by 0