# Information Bottleneck Approach to Spatial Attention Learning

arXiv: https://arxiv.org/abs/2108.03418 · HTML: https://mlsanity.com/paper/2108.03418/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 42 references that are in the map.
- First posted on arXiv 7 Aug 2021.

## Builds on (most cited first, 23 of 23)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [CBAM: Convolutional Block Attention Module](https://mlsanity.com/paper/1807.06521/) (2018)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [Deep Variational Information Bottleneck](https://mlsanity.com/paper/1612.00410/) (2016)
- [Attention-Based Models for Speech Recognition](https://mlsanity.com/paper/1506.07503/) (2015)
- [High Frequency Component Helps Explain the Generalization of Convolutional Neural Networks](https://mlsanity.com/paper/1905.13545/) (2019)
- [Local Relation Networks for Image Recognition](https://mlsanity.com/paper/1904.11491/) (2019)
- [Information Dropout: Learning Optimal Representations Through Noisy Computation](https://mlsanity.com/paper/1611.01353/) (2016)
- [Restricting the Flow: Information Bottlenecks for Attribution](https://mlsanity.com/paper/2001.00396/) (2020)
- [Attention Branch Network: Learning of Attention Mechanism for Visual Explanation](https://mlsanity.com/paper/1812.10025/) (2018)
- [Learning to Learn with Variational Information Bottleneck for Domain Generalization](https://mlsanity.com/paper/2007.07645/) (2020)
- [Learn To Pay Attention](https://mlsanity.com/paper/1804.02391/) (2018)
- [Action Recognition using Visual Attention](https://mlsanity.com/paper/1511.04119/) (2015)
- [Variational Attention for Sequence-to-Sequence Models](https://mlsanity.com/paper/1712.08207/) (2017)
- [InfoMask: Masked Variational Latent Representation to Localize Chest Disease](https://mlsanity.com/paper/1903.11741/) (2019)
- [Information-Bottleneck Approach to Salient Region Discovery](https://mlsanity.com/paper/1907.09578/) (2019)

## Built on by (most cited first, 6 of 6)

- [R2-Trans:Fine-Grained Visual Categorization with Redundancy Reduction](https://mlsanity.com/paper/2204.10095/) (2022)
- [Efficient Visual Transformer by Learnable Token Merging](https://mlsanity.com/paper/2407.15219/) (2024)
- [Recognizable Information Bottleneck](https://mlsanity.com/paper/2304.14618/) (2023)
- [Large Vision-Language Models Get Lost in Attention](https://mlsanity.com/paper/2605.05668/) (2026)
- [Learning Informative Attention Weights for Person Re-Identification](https://mlsanity.com/paper/2505.08961/) (2025)
- [Information Bottleneck-based Causal Attention for Multi-label Medical Image Recognition](https://mlsanity.com/paper/2508.08069/) (2025)

## Abstract (arXiv)

The selective visual attention mechanism in the human visual system (HVS) restricts the amount of information to reach visual awareness for perceiving natural scenes, allowing near real-time information processing with limited computational capacity [Koch and Ullman, 1987]. This kind of selectivity acts as an 'Information Bottleneck (IB)', which seeks a trade-off between information compression and predictive accuracy. However, such information constraints are rarely explored in the attention mechanism for deep neural networks (DNNs). In this paper, we propose an IB-inspired spatial attention module for DNN structures built for visual recognition. The module takes as input an intermediate representation of the input image, and outputs a variational 2D attention map that minimizes the mutual information (MI) between the attention-modulated representation and the input, while maximizing the MI between the attention-modulated representation and the task label. To further restrict the information bypassed by the attention map, we quantize the continuous attention scores to a set of learnable anchor values during training. Extensive experiments show that the proposed IB-inspired spatial attention mechanism can yield attention maps that neatly highlight the regions of interest while suppressing backgrounds, and bootstrap standard DNN structures for visual recognition tasks (e.g., image classification, fine-grained recognition, cross-domain classification). The attention maps are interpretable for the decision making of the DNNs as verified in the experiments. Our code is available at https://github.com/ashleylqx/AIB.git.