# Segmentation-Aware Convolutional Networks Using Local Attention Masks

arXiv: https://arxiv.org/abs/1708.04607 · HTML: https://mlsanity.com/paper/1708.04607/

## Key facts

- Cited by 31 papers in the map (data through 7 Oct 2026).
- First posted on arXiv 15 Aug 2017.

## Built on by (most cited first, 30 of 31)

- [Dynamic Neural Networks: A Survey](https://mlsanity.com/paper/2102.04906/) (2021)
- [Exploring Cross-Image Pixel Contrast for Semantic Segmentation](https://mlsanity.com/paper/2101.11939/) (2021)
- [Rethinking Semantic Segmentation: A Prototype View](https://mlsanity.com/paper/2203.15102/) (2022)
- [GMMSeg: Gaussian Mixture based Generative Semantic Segmentation Models](https://mlsanity.com/paper/2210.02025/) (2022)
- [Deep Hierarchical Semantic Segmentation](https://mlsanity.com/paper/2203.14335/) (2022)
- [RbA: Segmenting Unknown Regions Rejected by All](https://mlsanity.com/paper/2211.14293/) (2022)
- [Semantics for Robotic Mapping, Perception and Interaction: A Survey](https://mlsanity.com/paper/2101.00443/) (2021)
- [A Dynamic Multi-Scale Voxel Flow Network for Video Prediction](https://mlsanity.com/paper/2303.09875/) (2023)
- [Diverse Semantic Image Synthesis via Probability Distribution Modeling](https://mlsanity.com/paper/2103.06878/) (2021)
- [Stochastic Segmentation with Conditional Categorical Diffusion Models](https://mlsanity.com/paper/2303.08888/) (2023)
- [CLUSTSEG: Clustering for Universal Segmentation](https://mlsanity.com/paper/2305.02187/) (2023)
- [GaitGCI: Generative Counterfactual Intervention for Gait Recognition](https://mlsanity.com/paper/2306.03428/) (2023)
- [Dynamic Grained Encoder for Vision Transformers](https://mlsanity.com/paper/2301.03831/) (2023)
- [AerialFormer: Multi-resolution Transformer for Aerial Image Segmentation](https://mlsanity.com/paper/2306.06842/) (2023)
- [Background-Click Supervision for Temporal Action Localization](https://mlsanity.com/paper/2111.12449/) (2021)
- [LOGICSEG: Parsing Visual Semantics with Neural Logic Learning and Reasoning](https://mlsanity.com/paper/2309.13556/) (2023)
- [MetaGait: Learning to Learn an Omni Sample Adaptive Representation for Gait Recognition](https://mlsanity.com/paper/2306.03445/) (2023)
- [SSAN: Separable Self-Attention Network for Video Representation Learning](https://mlsanity.com/paper/2105.13033/) (2021)
- [GOSS: Towards Generalized Open-set Semantic Segmentation](https://mlsanity.com/paper/2203.12116/) (2022)
- [Theoretical Understanding of In-Context Learning in Shallow Transformers with Unstructured Data](https://mlsanity.com/paper/2402.00743/) (2024)
- [Structure-Informed Shadow Removal Networks](https://mlsanity.com/paper/2301.03182/) (2023)
- [Efficient Visual Recognition with Deep Neural Networks: A Survey on Recent Advances and New Directions](https://mlsanity.com/paper/2108.13055/) (2021)
- [Switchable Representation Learning Framework with Self-compatibility](https://mlsanity.com/paper/2206.08289/) (2022)
- [Latent Graph Attention for Enhanced Spatial Context](https://mlsanity.com/paper/2307.04149/) (2023)
- [Deep auxiliary learning for visual localization using colorization task](https://mlsanity.com/paper/2107.00222/) (2021)
- [Semantic Segmentation and Object Detection Towards Instance Segmentation: Breast Tumor Identification](https://mlsanity.com/paper/2108.03287/) (2021)
- [RelationRS: Relationship Representation Network for Object Detection in Aerial Images](https://mlsanity.com/paper/2110.06730/) (2021)
- [SCNet: A Generalized Attention-based Model for Crack Fault Segmentation](https://mlsanity.com/paper/2112.01426/) (2021)
- [Dual Complementary Dynamic Convolution for Image Recognition](https://mlsanity.com/paper/2211.06163/) (2022)
- [Deep-Learning-Assisted Analysis of Cataract Surgery Videos](https://mlsanity.com/paper/2312.05900/) (2023)

## Abstract (arXiv)

We introduce an approach to integrate segmentation information within a convolutional neural network (CNN). This counter-acts the tendency of CNNs to smooth information across regions and increases their spatial precision. To obtain segmentation information, we set up a CNN to provide an embedding space where region co-membership can be estimated based on Euclidean distance. We use these embeddings to compute a local attention mask relative to every neuron position. We incorporate such masks in CNNs and replace the convolution operation with a "segmentation-aware" variant that allows a neuron to selectively attend to inputs coming from its own region. We call the resulting network a segmentation-aware CNN because it adapts its filters at each image point according to local segmentation cues. We demonstrate the merit of our method on two widely different dense prediction tasks, that involve classification (semantic segmentation) and regression (optical flow). Our results show that in semantic segmentation we can match the performance of DenseCRFs while being faster and simpler, and in optical flow we obtain clearly sharper responses than networks that do not use local attention masks. In both cases, segmentation-aware convolution yields systematic improvements over strong baselines. Source code for this work is available online at http://cs.cmu.edu/~aharley/segaware.