# AttendSeg: A Tiny Attention Condenser Neural Network for Semantic Segmentation on the Edge

arXiv: https://arxiv.org/abs/2104.14623 · HTML: https://mlsanity.com/paper/2104.14623/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 47 references that are in the map.
- First posted on arXiv 29 Apr 2021.

## Builds on (most cited first, 30 of 37)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [CBAM: Convolutional Block Attention Module](https://mlsanity.com/paper/1807.06521/) (2018)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference](https://mlsanity.com/paper/1712.05877/) (2017)
- [ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices](https://mlsanity.com/paper/1707.01083/) (2017)
- [ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design](https://mlsanity.com/paper/1807.11164/) (2018)
- [SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size](https://mlsanity.com/paper/1602.07360/) (2016)
- [MnasNet: Platform-Aware Neural Architecture Search for Mobile](https://mlsanity.com/paper/1807.11626/) (2018)
- [BinaryConnect: Training Deep Neural Networks with binary weights during propagations](https://mlsanity.com/paper/1511.00363/) (2015)
- [FDA: Fourier Domain Adaptation for Semantic Segmentation](https://mlsanity.com/paper/2004.05498/) (2020)
- [RefineNet: Multi-Path Refinement Networks for High-Resolution Semantic Segmentation](https://mlsanity.com/paper/1611.06612/) (2016)
- [Understanding Convolution for Semantic Segmentation](https://mlsanity.com/paper/1702.08502/) (2017)
- [Single-Stage Semantic Segmentation from Image Labels](https://mlsanity.com/paper/2005.08104/) (2020)
- [Learning Dynamic Routing for Semantic Segmentation](https://mlsanity.com/paper/2003.10401/) (2020)
- [NetScore: Towards Universal Metrics for Large-scale Performance Analysis of Deep Neural Networks for Practical On-Device Edge Usage](https://mlsanity.com/paper/1806.05512/) (2018)
- [FermiNets: Learning generative machines to generate efficient neural networks via generative synthesis](https://mlsanity.com/paper/1809.05989/) (2018)
- [Task Transfer by Preference-Based Cost Learning](https://mlsanity.com/paper/1805.04686/) (2018)
- [TinySpeech: Attention Condensers for Deep Speech Recognition Neural Networks on Edge Devices](https://mlsanity.com/paper/2008.04245/) (2020)
- [ProjectionNet: Learning Efficient On-Device Deep Networks Using Neural Projections](https://mlsanity.com/paper/1708.00630/) (2017)

## Built on by (most cited first, 4 of 4)

- [Machine Learning for Microcontroller-Class Hardware: A Review](https://mlsanity.com/paper/2205.14550/) (2022)
- [Faster Attention Is What You Need: A Fast Self-Attention Neural Network Backbone Architecture for the Edge via Double-Condensing Attention Condensers](https://mlsanity.com/paper/2208.06980/) (2022)
- [L^3U-net: Low-Latency Lightweight U-net Based Image Segmentation Model for Parallel CNN Processors](https://mlsanity.com/paper/2203.16528/) (2022)
- [FMAS: Fast Multi-Objective SuperNet Architecture Search for Semantic Segmentation](https://mlsanity.com/paper/2303.16322/) (2023)

## Abstract (arXiv)

In this study, we introduce AttendSeg, a low-precision, highly compact deep neural network tailored for on-device semantic segmentation. AttendSeg possesses a self-attention network architecture comprising of light-weight attention condensers for improved spatial-channel selective attention at a very low complexity. The unique macro-architecture and micro-architecture design properties of AttendSeg strike a strong balance between representational power and efficiency, achieved via a machine-driven design exploration strategy tailored specifically for the task at hand. Experimental results demonstrated that the proposed AttendSeg can achieve segmentation accuracy comparable to much larger deep neural networks with greater complexity while possessing a significantly lower architecture and computational complexity (requiring as much as >27x fewer MACs, >72x fewer parameters, and >288x lower weight memory requirements), making it well-suited for TinyML applications on the edge.