# Mix-Pooling Strategy for Attention Mechanism

arXiv: https://arxiv.org/abs/2208.10322 · HTML: https://mlsanity.com/paper/2208.10322/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 34 references that are in the map.
- First posted on arXiv 22 Aug 2022.

## Builds on (most cited first, 25 of 25)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [CBAM: Convolutional Block Attention Module](https://mlsanity.com/paper/1807.06521/) (2018)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [Dual Attention Network for Scene Segmentation](https://mlsanity.com/paper/1809.02983/) (2018)
- [ECA-Net: Efficient Channel Attention for Deep Convolutional Neural Networks](https://mlsanity.com/paper/1910.03151/) (2019)
- [Residual Attention Network for Image Classification](https://mlsanity.com/paper/1704.06904/) (2017)
- [Recurrent Models of Visual Attention](https://mlsanity.com/paper/1406.6247/) (2014)
- [Attention Mechanisms in Computer Vision: A Survey](https://mlsanity.com/paper/2111.07624/) (2021)
- [DRAW: A Recurrent Neural Network For Image Generation](https://mlsanity.com/paper/1502.04623/) (2015)
- [FcaNet: Frequency Channel Attention Networks](https://mlsanity.com/paper/2012.11879/) (2020)
- [SRM : A Style-based Recalibration Module for Convolutional Neural Networks](https://mlsanity.com/paper/1903.10829/) (2019)
- [Gated Channel Transformation for Visual Recognition](https://mlsanity.com/paper/1909.11519/) (2019)
- [DIANet: Dense-and-Implicit Attention Network](https://mlsanity.com/paper/1905.10671/) (2019)
- [Spatial Group-wise Enhance: Improving Semantic Feature Learning in Convolutional Networks](https://mlsanity.com/paper/1905.09646/) (2019)
- [Convolution-Weight-Distribution Assumption: Rethinking the Criteria of Channel Pruning](https://mlsanity.com/paper/2004.11627/) (2020)
- [Instance Enhancement Batch Normalization: an Adaptive Regulator of Batch Noise](https://mlsanity.com/paper/1908.04008/) (2019)
- ["Garbage In, Garbage Out" Revisited: What Do Machine Learning Application Papers Report About Human-Labeled Training Data?](https://mlsanity.com/paper/2107.02278/) (2021)
- [Blending Pruning Criteria for Convolutional Neural Networks](https://mlsanity.com/paper/2107.05033/) (2021)
- [Real-World Image Super-Resolution by Exclusionary Dual-Learning](https://mlsanity.com/paper/2206.02609/) (2022)
- [Efficient Attention Network: Accelerate Attention by Searching Where to Plug](https://mlsanity.com/paper/2011.14058/) (2020)
- [Difficulty-aware Image Super Resolution via Deep Adaptive Dual-Network](https://mlsanity.com/paper/1904.05802/) (2019)
- [Stochastic Region Pooling: Make Attention More Expressive](https://mlsanity.com/paper/1904.09853/) (2019)

## Built on by (most cited first, 3 of 3)

- [A Generic Shared Attention Mechanism for Various Backbone Neural Networks](https://mlsanity.com/paper/2210.16101/) (2022)
- [LSAS: Lightweight Sub-attention Strategy for Alleviating Attention Bias Problem](https://mlsanity.com/paper/2305.05200/) (2023)
- [ASR: Attention-alike Structural Re-parameterization](https://mlsanity.com/paper/2304.06345/) (2023)

## Abstract (arXiv)

Recently many effective attention modules are proposed to boot the model performance by exploiting the internal information of convolutional neural networks in computer vision. In general, many previous works ignore considering the design of the pooling strategy of the attention mechanism since they adopt the global average pooling for granted, which hinders the further improvement of the performance of the attention mechanism. However, we empirically find and verify a phenomenon that the simple linear combination of global max-pooling and global min-pooling can produce pooling strategies that match or exceed the performance of global average pooling. Based on this empirical observation, we propose a simple-yet-effective attention module SPEM, which adopts a self-adaptive pooling strategy based on global max-pooling and global min-pooling and a lightweight module for producing the attention map. The effectiveness of SPEM is demonstrated by extensive experiments on widely-used benchmark datasets and popular attention networks.