# Region Mutual Information Loss for Semantic Segmentation

arXiv: https://arxiv.org/abs/1910.12037 · HTML: https://mlsanity.com/paper/1910.12037/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Oct 2019.

## Built on by (most cited first, 30 of 49)

- [Exploring Cross-Image Pixel Contrast for Semantic Segmentation](https://mlsanity.com/paper/2101.11939/) (2021)
- [Large-scale Unsupervised Semantic Segmentation](https://mlsanity.com/paper/2106.03149/) (2021)
- [Mining Contextual Information Beyond Image for Semantic Segmentation](https://mlsanity.com/paper/2108.11819/) (2021)
- [Loss Functions in the Era of Semantic Segmentation: A Survey and Outlook](https://mlsanity.com/paper/2312.05391/) (2023)
- [AutoGluon-Multimodal (AutoMM): Supercharging Multimodal AutoML with Foundation Models](https://mlsanity.com/paper/2404.16233/) (2024)
- [Omni-supervised Point Cloud Segmentation via Gradual Receptive Field Component Reasoning](https://mlsanity.com/paper/2105.10203/) (2021)
- [Boosting Semi-supervised Image Segmentation with Global and Local Mutual Information Regularization](https://mlsanity.com/paper/2103.04813/) (2021)
- [Where and How: Mitigating Confusion in Neural Radiance Fields from Sparse Inputs](https://mlsanity.com/paper/2308.02908/) (2023)
- [MCIBI++: Soft Mining Contextual Information Beyond Image for Semantic Segmentation](https://mlsanity.com/paper/2209.04471/) (2022)
- [PP-HumanSeg: Connectivity-Aware Portrait Segmentation with a Large-Scale Teleconferencing Video Dataset](https://mlsanity.com/paper/2112.07146/) (2021)
- [Enhancing Boundary Segmentation for Topological Accuracy with Skeleton-based Methods](https://mlsanity.com/paper/2404.18539/) (2024)
- [CitySurfaces: City-Scale Semantic Segmentation of Sidewalk Materials](https://mlsanity.com/paper/2201.02260/) (2022)
- [Multi-scale and Cross-scale Contrastive Learning for Semantic Segmentation](https://mlsanity.com/paper/2203.13409/) (2022)
- [Sentence Attention Blocks for Answer Grounding](https://mlsanity.com/paper/2309.11593/) (2023)
- [Warp-Refine Propagation: Semi-Supervised Auto-labeling via Cycle-consistency](https://mlsanity.com/paper/2109.13432/) (2021)
- [GOSS: Towards Generalized Open-set Semantic Segmentation](https://mlsanity.com/paper/2203.12116/) (2022)
- [PLGAN: Generative Adversarial Networks for Power-Line Segmentation in Aerial Images](https://mlsanity.com/paper/2204.07243/) (2022)
- [IDRNet: Intervention-Driven Relation Network for Semantic Segmentation](https://mlsanity.com/paper/2310.10755/) (2023)
- [Positive-Negative Equal Contrastive Loss for Semantic Segmentation](https://mlsanity.com/paper/2207.01417/) (2022)
- [Soft labelling for semantic segmentation: Bringing coherence to label down-sampling](https://mlsanity.com/paper/2302.13961/) (2023)
- [Accelerating Neural Network Training with Distributed Asynchronous and Selective Optimization (DASO)](https://mlsanity.com/paper/2104.05588/) (2021)
- [Dense Prediction with Attentive Feature Aggregation](https://mlsanity.com/paper/2111.00770/) (2021)
- [MetaMorph: Learning Metamorphic Image Transformation With Appearance Changes](https://mlsanity.com/paper/2303.04849/) (2023)
- [SFDA-rPPG: Source-Free Domain Adaptive Remote Physiological Measurement with Spatio-Temporal Consistency](https://mlsanity.com/paper/2409.12040/) (2024)
- [Semantic-Aware Contrastive Learning for Multi-object Medical Image Segmentation](https://mlsanity.com/paper/2106.01596/) (2021)
- [Self-adversarial Multi-scale Contrastive Learning for Semantic Segmentation of Thermal Facial Images](https://mlsanity.com/paper/2209.10700/) (2022)
- [Semantic Segmentation with Active Semi-Supervised Representation Learning](https://mlsanity.com/paper/2210.08403/) (2022)
- [Hard Exudate Segmentation Supplemented by Super-Resolution with Multi-scale Attention Fusion Module](https://mlsanity.com/paper/2211.09404/) (2022)
- [QuadFormer: Quadruple Transformer for Unsupervised Domain Adaptation in Power Line Segmentation of Aerial Images](https://mlsanity.com/paper/2211.16988/) (2022)
- [Constrained Multiview Representation for Self-supervised Contrastive Learning](https://mlsanity.com/paper/2402.03456/) (2024)

## Abstract (arXiv)

Semantic segmentation is a fundamental problem in computer vision. It is considered as a pixel-wise classification problem in practice, and most segmentation models use a pixel-wise loss as their optimization riterion. However, the pixel-wise loss ignores the dependencies between pixels in an image. Several ways to exploit the relationship between pixels have been investigated, eg, conditional random fields (CRF) and pixel affinity based methods. Nevertheless, these methods usually require additional model branches, large extra memories, or more inference time. In this paper, we develop a region mutual information (RMI) loss to model the dependencies among pixels more simply and efficiently. In contrast to the pixel-wise loss which treats the pixels as independent samples, RMI uses one pixel and its neighbour pixels to represent this pixel. Then for each pixel in an image, we get a multi-dimensional point that encodes the relationship between pixels, and the image is cast into a multi-dimensional distribution of these high-dimensional points. The prediction and ground truth thus can achieve high order consistency through maximizing the mutual information (MI) between their multi-dimensional distributions. Moreover, as the actual value of the MI is hard to calculate, we derive a lower bound of the MI and maximize the lower bound to maximize the real value of the MI. RMI only requires a few extra computational resources in the training stage, and there is no overhead during testing. Experimental results demonstrate that RMI can achieve substantial and consistent improvements in performance on PASCAL VOC 2012 and CamVid datasets. The code is available at https://github.com/ZJULearning/RMI.