mlsanityData to 7 Oct 2026

A Lightweight and Gradient-Stable Neural Layer

arXiv 2106.04088 ↗First posted 8 Jun 2021cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

To enhance resource efficiency and model deployability of neural networks, we propose a neural-layer architecture based on Householder weighting and absolute-value activating, called Householder-absolute neural layer or simply Han-layer. Compared to a fully connected layer with d-neurons and d outputs, a Han-layer reduces the number of parameters and the corresponding computational complexity from O(d2) to O(d). The Han-layer structure guarantees that the Jacobian of the layer function is always orthogonal, thus ensuring gradient stability (i.e., free of gradient vanishing or exploding issues) for any Han-layer sub-networks. Extensive numerical experiments show that one can strategically use Han-layers to replace fully connected (FC) layers, reducing the number of model parameters while maintaining or even improving the generalization performance. We will also showcase the capabilities of the Han-layer architecture on a few small stylized models, and discuss its current limitations.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 36 of its 60 references that are in the map. First posted on arXiv 8 Jun 2021.

Builds on

36 of 60 references.

References · 36

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  5. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  6. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  7. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,976
  8. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  9. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,623
  10. 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,510
  11. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  12. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
24 more
  1. 2021Transformers in Vision: A Surveycited by 1,009
  2. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 935
  3. 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 541
  4. 2017A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasetscited by 374
  5. 2021Pay Attention to MLPscited by 367
  6. 2015Unitary Evolution Recurrent Neural Networkscited by 291
  7. 2021Pruning and Quantization for Deep Neural Network Acceleration: A Surveycited by 259
  8. 2018Sorting out Lipschitz function approximationcited by 211
  9. 2018Lipschitz-Margin Training: Scalable Certification of Perturbation Invariance for Deep Neural Networkscited by 193
  10. 2018Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networkscited by 176
  11. 2015A Simple Way to Initialize Recurrent Networks of Rectified Linear Unitscited by 130
  12. 2019Orthogonal Convolutional Neural Networkscited by 122
  13. 2016Neural Photo Editing with Introspective Adversarial Networkscited by 109
  14. 2017Orthogonal Weight Normalization: Solution to Optimization over Multiple Dependent Stiefel Manifolds in Deep Neural Networkscited by 92
  15. 2016Full-Capacity Unitary Recurrent Neural Networkscited by 90
  16. 2017On orthogonality and learning recurrent networks with long term dependenciescited by 82
  17. 2018The Emergence of Spectral Universality in Deep Networkscited by 74
  18. 2020Provable Benefit of Orthogonal Initialization in Optimizing Deep Linear Networkscited by 73
  19. 2019Orthogonal Deep Neural Networkscited by 72
  20. 2016Efficient Orthogonal Parametrisation of Recurrent Neural Networks Using Householder Reflectionscited by 56
  21. 2016Improving Variational Auto-Encoders using Householder Flowcited by 41
  22. 2018Stabilizing Gradients for Deep Neural Networks via Efficient SVD Parameterizationcited by 35
  23. 2018Norm-Preservation: Why Residual Networks Can Become Extremely Deep?cited by 12
  24. 2019Efficient hinging hyperplanes neural network and its application in nonlinear system identificationcited by 4

Built on by

Not cited yet.