# Neural Architecture Search for Lightweight Non-Local Networks

arXiv: https://arxiv.org/abs/2004.01961 · HTML: https://mlsanity.com/paper/2004.01961/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Apr 2020.

## Built on by (most cited first, 23 of 23)

- [Visformer: The Vision-friendly Transformer](https://mlsanity.com/paper/2104.12533/) (2021)
- [RobustART: Benchmarking Robustness on Architecture Design and Training Techniques](https://mlsanity.com/paper/2109.05211/) (2021)
- [A Comprehensive Survey on Hardware-Aware Neural Architecture Search](https://mlsanity.com/paper/2101.09336/) (2021)
- [CMT-DeepLab: Clustering Mask Transformers for Panoptic Segmentation](https://mlsanity.com/paper/2206.08948/) (2022)
- [GhostNetV2: Enhance Cheap Operation with Long-Range Attention](https://mlsanity.com/paper/2211.12905/) (2022)
- [Prototypical Cross-Attention Networks for Multiple Object Tracking and Segmentation](https://mlsanity.com/paper/2106.11958/) (2021)
- [Teachers Do More Than Teach: Compressing Image-to-Image Models](https://mlsanity.com/paper/2103.03467/) (2021)
- [Augmenting Deep Classifiers with Polynomial Neural Networks](https://mlsanity.com/paper/2104.07916/) (2021)
- [Self-Regulation for Semantic Segmentation](https://mlsanity.com/paper/2108.09702/) (2021)
- [Video Salient Object Detection via Contrastive Features and Attention Modules](https://mlsanity.com/paper/2111.02368/) (2021)
- [SPFormer: Enhancing Vision Transformer with Superpixel Representation](https://mlsanity.com/paper/2401.02931/) (2024)
- [kMaX-DeepLab: k-means Mask Transformer](https://mlsanity.com/paper/2207.04044/) (2022)
- [Contrastive Self-supervised Neural Architecture Search](https://mlsanity.com/paper/2102.10557/) (2021)
- [VMNet: Voxel-Mesh Network for Geodesic-Aware 3D Semantic Segmentation](https://mlsanity.com/paper/2107.13824/) (2021)
- [Searching for TrioNet: Combining Convolution with Local and Global Self-Attention](https://mlsanity.com/paper/2111.07547/) (2021)
- [Rethinking Efficacy of Softmax for Lightweight Non-Local Neural Networks](https://mlsanity.com/paper/2207.13423/) (2022)
- [Cross Task Neural Architecture Search for EEG Signal Classifications](https://mlsanity.com/paper/2210.06298/) (2022)
- [Salient Positions based Attention Network for Image Classification](https://mlsanity.com/paper/2106.04996/) (2021)
- [Semiconductor Defect Pattern Classification by Self-Proliferation-and-Attention Neural Network](https://mlsanity.com/paper/2212.00345/) (2022)
- [NEON: Enabling Efficient Support for Nonlinear Operations in Resistive RAM-based Neural Network Accelerators](https://mlsanity.com/paper/2211.05730/) (2022)
- [Miniaturized Graph Convolutional Networks with Topologically Consistent Pruning](https://mlsanity.com/paper/2306.17590/) (2023)
- [RepNet-VSR: Reparameterizable Architecture for High-Fidelity Video Super-Resolution](https://mlsanity.com/paper/2504.15649/) (2025)
- [Learning Informative Attention Weights for Person Re-Identification](https://mlsanity.com/paper/2505.08961/) (2025)

## Abstract (arXiv)

Non-Local (NL) blocks have been widely studied in various vision tasks. However, it has been rarely explored to embed the NL blocks in mobile neural networks, mainly due to the following challenges: 1) NL blocks generally have heavy computation cost which makes it difficult to be applied in applications where computational resources are limited, and 2) it is an open problem to discover an optimal configuration to embed NL blocks into mobile neural networks. We propose AutoNL to overcome the above two obstacles. Firstly, we propose a Lightweight Non-Local (LightNL) block by squeezing the transformation operations and incorporating compact features. With the novel design choices, the proposed LightNL block is 400x computationally cheaper} than its conventional counterpart without sacrificing the performance. Secondly, by relaxing the structure of the LightNL block to be differentiable during training, we propose an efficient neural architecture search algorithm to learn an optimal configuration of LightNL blocks in an end-to-end manner. Notably, using only 32 GPU hours, the searched AutoNL model achieves 77.7% top-1 accuracy on ImageNet under a typical mobile setting (350M FLOPs), significantly outperforming previous mobile models including MobileNetV2 (+5.7%), FBNet (+2.8%) and MnasNet (+2.1%). Code and models are available at https://github.com/LiYingwei/AutoNL.