HDAM: Heuristic Difference Attention Module for Convolutional Neural Networks
Not on the map. Why?
Abstract (arXiv)
The attention mechanism is one of the most important priori knowledge to enhance convolutional neural networks. Most attention mechanisms are bound to the convolutional layer and use local or global contextual information to recalibrate the input. This is a popular attention strategy design method. Global contextual information helps the network to consider the overall distribution, while local contextual information is more general. The contextual information makes the network pay attention to the mean or maximum value of a particular receptive field. Different from the most attention mechanism, this article proposes a novel attention mechanism with the heuristic difference attention module, HDAM. HDAM's input recalibration is based on the difference between the local and global contextual information instead of the mean and maximum values. At the same time, to make different layers have a more suitable local receptive field size and increase the exibility of the local receptive field design, we use genetic algorithm to heuristically produce local receptive fields. First, HDAM extracts the mean value of the global and local receptive fields as the corresponding contextual information. Then the difference between the global and local contextual information is calculated. Finally HDAM uses this difference to recalibrate the input. In addition, we use the heuristic ability of genetic algorithm to search for the local receptive field size of each layer. Our experiments on CIFAR-10 and CIFAR-100 show that HDAM can use fewer parameters than other attention mechanisms to achieve higher accuracy. We implement HDAM with the Python library, Pytorch, and the code and models will be publicly available.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 27 of its 39 references that are in the map. First posted on arXiv 19 Feb 2022.
Builds on
27 of 39 references.
References · 27
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
- 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
- 2016Densely Connected Convolutional Networkscited by 6,720
- 2017Squeeze-and-Excitation Networkscited by 4,789
- 2014Going Deeper with Convolutionscited by 4,241
- 2016Wide Residual Networkscited by 3,070
- 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,546
- 2018CBAM: Convolutional Block Attention Modulecited by 2,477
- 2016Xception: Deep Learning with Depthwise Separable Convolutionscited by 2,332
- 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,628
- 2016Neural Architecture Search with Reinforcement Learningcited by 1,499
15 more
- 2014Striving for Simplicity: The All Convolutional Netcited by 990
- 2013Network In Networkcited by 605
- 2018Deep Learning for Generic Object Detection: A Surveycited by 392
- 2017Large-Scale Evolution of Image Classifierscited by 348
- 2016Designing Neural Network Architectures using Reinforcement Learningcited by 279
- 2015Highway Networkscited by 260
- 2016FractalNet: Ultra-Deep Neural Networks without Residualscited by 216
- 2013Maxout Networkscited by 187
- 2018BAM: Bottleneck Attention Modulecited by 172
- 2020A Survey on Evolutionary Neural Architecture Searchcited by 142
- 2018Gather-Excite: Exploiting Feature Context in Convolutional Neural Networkscited by 139
- 2017Genetic CNNcited by 124
- 2017A Genetic Programming Approach to Designing Convolutional Neural Network Architecturescited by 59
- 2020Evolution of Image Segmentation using Deep Convolutional Neural Network: A Surveycited by 28
- 2021"BNN - BN = ?": Training Binary Neural Networks without Batch Normalizationcited by 17
Built on by
Not cited yet.