# Disentangled Non-Local Neural Networks

arXiv: https://arxiv.org/abs/2006.06668 · HTML: https://mlsanity.com/paper/2006.06668/

## Key facts

- Cited by 132 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Jun 2020.

## Built on by (most cited first, 30 of 132)

- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Video Swin Transformer](https://mlsanity.com/paper/2106.13230/) (2021)
- [Segmenter: Transformer for Semantic Segmentation](https://mlsanity.com/paper/2105.05633/) (2021)
- [DenseCLIP: Language-Guided Dense Prediction with Context-Aware Prompting](https://mlsanity.com/paper/2112.01518/) (2021)
- [Focal Self-attention for Local-Global Interactions in Vision Transformers](https://mlsanity.com/paper/2107.00641/) (2021)
- [Transformers in Medical Imaging: A Survey](https://mlsanity.com/paper/2201.09873/) (2022)
- [Exploring Cross-Image Pixel Contrast for Semantic Segmentation](https://mlsanity.com/paper/2101.11939/) (2021)
- [Attention Mechanisms in Computer Vision: A Survey](https://mlsanity.com/paper/2111.07624/) (2021)
- [CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation with Transformers](https://mlsanity.com/paper/2203.04838/) (2022)
- [A Simple Baseline for Open-Vocabulary Semantic Segmentation with Pre-trained Vision-language Model](https://mlsanity.com/paper/2112.14757/) (2021)
- [K-Net: Towards Unified Image Segmentation](https://mlsanity.com/paper/2106.14855/) (2021)
- [CycleMLP: A MLP-like Architecture for Dense Prediction](https://mlsanity.com/paper/2107.10224/) (2021)
- [AS-MLP: An Axial Shifted MLP Architecture for Vision](https://mlsanity.com/paper/2107.08391/) (2021)
- [Contextual Transformer Networks for Visual Recognition](https://mlsanity.com/paper/2107.12292/) (2021)
- [You Only Learn One Representation: Unified Network for Multiple Tasks](https://mlsanity.com/paper/2105.04206/) (2021)
- [Bending Reality: Distortion-aware Transformers for Adapting to Panoramic Semantic Segmentation](https://mlsanity.com/paper/2203.01452/) (2022)
- [Exploring Relational Context for Multi-Task Dense Prediction](https://mlsanity.com/paper/2104.13874/) (2021)
- [Multiple Object Tracking with Correlation Learning](https://mlsanity.com/paper/2104.03541/) (2021)
- [Safe Self-Refinement for Transformer-based Domain Adaptation](https://mlsanity.com/paper/2204.07683/) (2022)
- [RelationTrack: Relation-aware Multiple Object Tracking with Decoupled Representation](https://mlsanity.com/paper/2105.04322/) (2021)
- [Capturing Omni-Range Context for Omnidirectional Segmentation](https://mlsanity.com/paper/2103.05687/) (2021)
- [AAformer: Auto-Aligned Transformer for Person Re-Identification](https://mlsanity.com/paper/2104.00921/) (2021)
- [Mining Contextual Information Beyond Image for Semantic Segmentation](https://mlsanity.com/paper/2108.11819/) (2021)
- [Unveiling the Potential of Structure Preserving for Weakly Supervised Object Localization](https://mlsanity.com/paper/2103.04523/) (2021)
- [Behind Every Domain There is a Shift: Adapting Distortion-aware Vision Transformers for Panoramic Semantic Segmentation](https://mlsanity.com/paper/2207.11860/) (2022)
- [CTNet: Context-based Tandem Network for Semantic Segmentation](https://mlsanity.com/paper/2104.09805/) (2021)
- [MixFormer: Mixing Features across Windows and Dimensions](https://mlsanity.com/paper/2204.02557/) (2022)
- [Active Boundary Loss for Semantic Segmentation](https://mlsanity.com/paper/2102.02696/) (2021)
- [ISNet: Integrate Image-Level and Semantic-Level Context for Semantic Segmentation](https://mlsanity.com/paper/2108.12382/) (2021)
- [DensePASS: Dense Panoramic Semantic Segmentation via Unsupervised Domain Adaptation with Attention-Augmented Context Exchange](https://mlsanity.com/paper/2108.06383/) (2021)

## Abstract (arXiv)

The non-local block is a popular module for strengthening the context modeling ability of a regular convolutional neural network. This paper first studies the non-local block in depth, where we find that its attention computation can be split into two terms, a whitened pairwise term accounting for the relationship between two pixels and a unary term representing the saliency of every pixel. We also observe that the two terms trained alone tend to model different visual clues, e.g. the whitened pairwise term learns within-region relationships while the unary term learns salient boundaries. However, the two terms are tightly coupled in the non-local block, which hinders the learning of each. Based on these findings, we present the disentangled non-local block, where the two terms are decoupled to facilitate learning for both terms. We demonstrate the effectiveness of the decoupled design on various tasks, such as semantic segmentation on Cityscapes, ADE20K and PASCAL Context, object detection on COCO, and action recognition on Kinetics.