mlsanityData to 14 Oct 2026

GAAF: Searching Activation Functions for Binary Neural Networks through Genetic Algorithm

arXiv 2206.03291 ↗First posted 5 Jun 2022cs.NE cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

Binary neural networks (BNNs) show promising utilization in cost and power-restricted domains such as edge devices and mobile systems. This is due to its significantly less computation and storage demand, but at the cost of degraded performance. To close the accuracy gap, in this paper we propose to add a complementary activation function (AF) ahead of the sign based binarization, and rely on the genetic algorithm (GA) to automatically search for the ideal AFs. These AFs can help extract extra information from the input data in the forward pass, while allowing improved gradient approximation in the backward pass. Fifteen novel AFs are identified through our GA-based search, while most of them show improved performance (up to 2.54% on ImageNet) when testing on different datasets and network models. Our method offers a novel approach for designing general and application-specific BNN architecture. Our code is available at http://github.com/flying-Yan/GAAF.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 31 of its 53 references that are in the map. First posted on arXiv 5 Jun 2022.

Builds on

31 of 53 references.

References · 31

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  3. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  4. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  5. 2018DARTS: Differentiable Architecture Searchcited by 1,798
  6. 2016Neural Architecture Search with Reinforcement Learningcited by 1,499
  7. 2018Regularized Evolution for Image Classifier Architecture Searchcited by 909
  8. 2018ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardwarecited by 673
  9. 2020What is the State of Neural Network Pruning?cited by 583
  10. 2016DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradientscited by 569
  11. 2017Large-Scale Evolution of Image Classifierscited by 348
  12. 2018SNAS: Stochastic Neural Architecture Searchcited by 289
19 more
  1. 2016Designing Neural Network Architectures using Reinforcement Learningcited by 279
  2. 2018Bi-Real Net: Enhancing the Performance of 1-bit CNNs With Improved Representational Capability and Advanced Training Algorithmcited by 235
  3. 2018Neural Architecture Search with Bayesian Optimisation and Optimal Transportcited by 195
  4. 2020Binary Neural Networks: A Surveycited by 190
  5. 2020ReActNet: Towards Precise Binary Neural Network with Generalized Activation Functionscited by 184
  6. 2019Forward and Backward Information Retention for Accurate Binary Neural Networkscited by 169
  7. 2018Efficient Multi-objective Neural Architecture Search via Lamarckian Evolutioncited by 153
  8. 2017Towards Accurate Binary Convolutional Neural Networkcited by 149
  9. 2019Latent Weights Do Not Exist: Rethinking Binarized Neural Network Optimizationcited by 60
  10. 2016Loss-aware Binarization of Deep Networkscited by 56
  11. 2020Evolving Normalization-Activation Layerscited by 45
  12. 2017Attacking Binarized Neural Networkscited by 29
  13. 2018Efficient Super Resolution Using Binarized Neural Networkcited by 29
  14. 2019Circulant Binary Convolutional Networks: Enhancing the Performance of 1-bit DCNNs with Circulant Back Propagationcited by 28
  15. 2017The High-Dimensional Geometry of Binary Neural Networkscited by 22
  16. 2014Raiders of the Lost Architecture: Kernels for Bayesian Optimization in Conditional Parameter Spacescited by 16
  17. 2020Accelerating Binarized Neural Networks via Bit-Tensor-Cores in Turing GPUscited by 14
  18. 2019Self-Binarizing Networkscited by 11
  19. 2021BinaryCoP: Binary Neural Network-based COVID-19 Face-Mask Wear and Positioning Predictor on Edge Devicescited by 7

Built on by

Cited by year

Citing papers per year: 2024: 1.

1 paper.

  1. 2024Activation Function Optimization Scheme for Image Classificationcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024NAS-BNN: Neural Architecture Search for Binary Neural Networks
  2. 2022LAB: Learnable Activation Binarizer for Binary Neural Networks
  3. 2021Dynamic Binary Neural Network by learning channel-wise thresholds
  4. 2021A comprehensive review of Binary Neural Network
  5. 2022AdaBin: Improving Binary Neural Networks with Adaptive Binary Sets
  6. 2022Join the High Accuracy Club on ImageNet with A Binary Neural Network Ticket
  7. 2021A Bop and Beyond: A Second Order Optimizer for Binarized Neural Networks
  8. 2022Binarizing by Classification: Is soft function really necessary?