# Self-Regulation for Semantic Segmentation

arXiv: https://arxiv.org/abs/2108.09702 · HTML: https://mlsanity.com/paper/2108.09702/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 55 of its 73 references that are in the map.
- First posted on arXiv 22 Aug 2021.

## Builds on (most cited first, 30 of 55)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation](https://mlsanity.com/paper/1511.00561/) (2015)
- [Deep High-Resolution Representation Learning for Visual Recognition](https://mlsanity.com/paper/1908.07919/) (2019)
- [Multi-Scale Context Aggregation by Dilated Convolutions](https://mlsanity.com/paper/1511.07122/) (2015)
- [Dual Attention Network for Scene Segmentation](https://mlsanity.com/paper/1809.02983/) (2018)
- [Stacked Hourglass Networks for Human Pose Estimation](https://mlsanity.com/paper/1603.06937/) (2016)
- [Deformable ConvNets v2: More Deformable, Better Results](https://mlsanity.com/paper/1811.11168/) (2018)
- [BranchyNet: Fast Inference via Early Exiting from Deep Neural Networks](https://mlsanity.com/paper/1709.01686/) (2017)
- [Segmentation Transformer: Object-Contextual Representations for Semantic Segmentation](https://mlsanity.com/paper/1909.11065/) (2019)
- [Similarity-Preserving Knowledge Distillation](https://mlsanity.com/paper/1907.09682/) (2019)
- [Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation](https://mlsanity.com/paper/1905.08094/) (2019)
- [Res2Net: A New Multi-scale Backbone Architecture](https://mlsanity.com/paper/1904.01169/) (2019)
- [Deep Learning for Generic Object Detection: A Survey](https://mlsanity.com/paper/1809.02165/) (2018)
- [Learning Deconvolution Network for Semantic Segmentation](https://mlsanity.com/paper/1505.04366/) (2015)
- [GCNet: Non-local Networks Meet Squeeze-Excitation Networks and Beyond](https://mlsanity.com/paper/1904.11492/) (2019)
- [Self-supervised Equivariant Attention Mechanism for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2004.04581/) (2020)
- [Context Encoding for Semantic Segmentation](https://mlsanity.com/paper/1803.08904/) (2018)
- [Weakly Supervised Learning of Instance Segmentation with Inter-pixel Relations](https://mlsanity.com/paper/1904.05044/) (2019)
- [Causal Intervention for Weakly-Supervised Semantic Segmentation](https://mlsanity.com/paper/2009.12547/) (2020)

## Built on by (most cited first, 18 of 18)

- [Learning Local and Global Temporal Contexts for Video Semantic Segmentation](https://mlsanity.com/paper/2204.03330/) (2022)
- [SelfReg-UNet: Self-Regularized UNet for Medical Image Segmentation](https://mlsanity.com/paper/2406.14896/) (2024)
- [Parameter Efficient Fine-tuning via Cross Block Orchestration for Segment Anything Model](https://mlsanity.com/paper/2311.17112/) (2023)
- [Centralized Feature Pyramid for Object Detection](https://mlsanity.com/paper/2210.02093/) (2022)
- [Semantic Scene Completion with Cleaner Self](https://mlsanity.com/paper/2303.09977/) (2023)
- [Continual Learning for Image Segmentation with Dynamic Query](https://mlsanity.com/paper/2311.17450/) (2023)
- [Graph Reasoning Transformer for Image Parsing](https://mlsanity.com/paper/2209.09545/) (2022)
- [Understanding the Tricks of Deep Learning in Medical Image Segmentation: Challenges and Future Directions](https://mlsanity.com/paper/2209.10307/) (2022)
- [Coupling Global Context and Local Contents for Weakly-Supervised Semantic Segmentation](https://mlsanity.com/paper/2304.09059/) (2023)
- [Boosting Weakly-Supervised Image Segmentation via Representation, Transform, and Compensator](https://mlsanity.com/paper/2309.00871/) (2023)
- [A Recover-then-Discriminate Framework for Robust Anomaly Detection](https://mlsanity.com/paper/2406.04608/) (2024)
- [Rethinking Boundary Detection in Deep Learning-Based Medical Image Segmentation](https://mlsanity.com/paper/2505.04652/) (2025)
- [Learning to Reduce Information Bottleneck for Object Detection in Aerial Images](https://mlsanity.com/paper/2204.02033/) (2022)
- [Generalized Task-Driven Medical Image Quality Enhancement with Gradient Promotion](https://mlsanity.com/paper/2501.01114/) (2025)
- [Towards Improving Workers' Safety and Progress Monitoring of Construction Sites Through Construction Site Understanding](https://mlsanity.com/paper/2210.15760/) (2022)
- [Synthetic Instance Segmentation from Semantic Image Segmentation Masks](https://mlsanity.com/paper/2308.00949/) (2023)
- [Towards Customized Knowledge Distillation for Chip-Level Dense Image Predictions](https://mlsanity.com/paper/2401.13174/) (2024)
- [Contextrast++: Robust Multi-Scale Contextual Contrastive Learning for Semantic Segmentation](https://mlsanity.com/paper/2608.22679/) (2026)

## Abstract (arXiv)

In this paper, we seek reasons for the two major failure cases in Semantic Segmentation (SS): 1) missing small objects or minor object parts, and 2) mislabeling minor parts of large objects as wrong classes. We have an interesting finding that Failure-1 is due to the underuse of detailed features and Failure-2 is due to the underuse of visual contexts. To help the model learn a better trade-off, we introduce several Self-Regulation (SR) losses for training SS neural networks. By "self", we mean that the losses are from the model per se without using any additional data or supervision. By applying the SR losses, the deep layer features are regulated by the shallow ones to preserve more details; meanwhile, shallow layer classification logits are regulated by the deep ones to capture more semantics. We conduct extensive experiments on both weakly and fully supervised SS tasks, and the results show that our approach consistently surpasses the baselines. We also validate that SR losses are easy to implement in various state-of-the-art SS models, e.g., SPGNet and OCRNet, incurring little computational overhead during training and none for testing.