mlsanityData to 14 Oct 2026

RepAct: The Re-parameterizable Adaptive Activation Function

arXiv 2407.00131 ↗First posted 28 Jun 2024cs.LG cs.AI cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Addressing the imperative need for efficient artificial intelligence in IoT and edge computing, this study presents RepAct, a re-parameterizable adaptive activation function tailored for optimizing lightweight neural networks within the computational limitations of edge devices. By employing a multi-branch structure with learnable adaptive weights, RepAct enriches feature processing and enhances cross-layer interpretability. When evaluated on tasks such as image classification and object detection, RepAct notably surpassed conventional activation functions in lightweight networks, delivering up to a 7.92% accuracy boost on MobileNetV3-Small for the ImageNet100 dataset, while maintaining computational complexity on par with HardSwish. This innovative approach not only maximizes model parameter efficiency but also significantly improves the performance and understanding capabilities of lightweight neural networks, demonstrating its potential for real-time edge computing applications.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 36 of its 51 references that are in the map. First posted on arXiv 28 Jun 2024.

Builds on

36 of 51 references.

References · 36

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  4. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  5. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  6. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  7. 2017Squeeze-and-Excitation Networkscited by 4,789
  8. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  9. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  10. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  11. 2017Rethinking Atrous Convolution for Semantic Image Segmentationcited by 2,161
  12. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
24 more
  1. 2019Searching for MobileNetV3cited by 1,994
  2. 2018ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Designcited by 1,136
  3. 2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,070
  4. 2018Efficient Neural Architecture Search via Parameter Sharingcited by 837
  5. 2017Channel Pruning for Accelerating Very Deep Neural Networkscited by 702
  6. 2017Self-Normalizing Neural Networkscited by 598
  7. 2021RepVGG: Making VGG-style ConvNets Great Againcited by 522
  8. 2015Empirical Evaluation of Rectified Activations in Convolutional Networkcited by 519
  9. 2017ThiNet: A Filter Level Pruning Method for Deep Neural Network Compressioncited by 442
  10. 2019ACNet: Strengthening the Kernel Skeletons for Powerful CNN via Asymmetric Convolution Blockscited by 178
  11. 2021Activation Functions in Deep Learning: A Comprehensive Survey and Benchmarkcited by 155
  12. 2023RepViT: Revisiting Mobile CNN From ViT Perspectivecited by 123
  13. 2022MobileOne: An Improved One millisecond Mobile Backbonecited by 109
  14. 2021Diverse Branch Block: Building a Convolution as an Inception-like Unitcited by 87
  15. 2017Continuously Differentiable Exponential Linear Unitscited by 73
  16. 2020Dynamic ReLUcited by 55
  17. 2022Re-parameterizing Your Optimizers rather than Architecturescited by 35
  18. 2021RepMLPNet: Hierarchical Vision MLP with Re-parameterized Localitycited by 32
  19. 2023VanillaNet: the Power of Minimalism in Deep Learningcited by 23
  20. 2020Activate or Not: Learning Customized Activationcited by 22
  21. 2020AReLU: Attention-based Rectified Linear Unitcited by 9
  22. 2020Funnel Activation for Visual Recognitioncited by 9
  23. 2020ALReLU: A different approach on Leaky ReLU activation function to improve Neural Networks Performancecited by 7
  24. 2023Any Deep ReLU Network is Shallowcited by 6

Built on by

Not cited yet.