# Reducing Unimodal Bias in Multi-Modal Semantic Segmentation with Multi-Scale Functional Entropy Regularization

arXiv: https://arxiv.org/abs/2505.06635 · HTML: https://mlsanity.com/paper/2505.06635/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 55 of its 78 references that are in the map.
- First posted on arXiv 10 May 2025.

## Builds on (most cited first, 30 of 55)

- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [MultiMAE: Multi-modal Multi-task Masked Autoencoders](https://mlsanity.com/paper/2204.01678/) (2022)
- [CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation with Transformers](https://mlsanity.com/paper/2203.04838/) (2022)
- [DeepFusion: Lidar-Camera Deep Fusion for Multi-Modal 3D Object Detection](https://mlsanity.com/paper/2203.08195/) (2022)
- [2DPASS: 2D Priors Assisted Semantic Segmentation on LiDAR Point Clouds](https://mlsanity.com/paper/2207.04397/) (2022)
- [Multimodal Token Fusion for Vision Transformers](https://mlsanity.com/paper/2204.08721/) (2022)
- [Multimodal Prompting with Missing Modalities for Visual Recognition](https://mlsanity.com/paper/2303.03369/) (2023)
- [Deep Multimodal Fusion by Channel Exchanging](https://mlsanity.com/paper/2011.05005/) (2020)
- [Delivering Arbitrary-Modal Semantic Segmentation](https://mlsanity.com/paper/2303.01480/) (2023)
- [EV-SegNet: Semantic Segmentation for Event-based Cameras](https://mlsanity.com/paper/1811.12039/) (2018)
- [Deep Learning for Event-based Vision: A Comprehensive Survey and Benchmarks](https://mlsanity.com/paper/2302.08890/) (2023)
- [PST900: RGB-Thermal Calibration, Dataset and Segmentation Network](https://mlsanity.com/paper/1909.10980/) (2019)
- [ShapeConv: Shape-aware Convolutional Layer for Indoor RGB-D Semantic Segmentation](https://mlsanity.com/paper/2108.10528/) (2021)
- [MSeg3D: Multi-modal 3D Semantic Segmentation for Autonomous Driving](https://mlsanity.com/paper/2303.08600/) (2023)
- [Removing Bias in Multi-modal Classifiers: Regularization by Maximizing Functional Entropies](https://mlsanity.com/paper/2010.10802/) (2020)
- [CamLiFlow: Bidirectional Camera-LiDAR Fusion for Joint Optical Flow and Scene Flow Estimation](https://mlsanity.com/paper/2111.10502/) (2021)
- [UniBind: LLM-Augmented Unified and Balanced Representation Space to Bind Them All](https://mlsanity.com/paper/2403.12532/) (2024)
- [Spatial Information Guided Convolution for Real-Time RGBD Semantic Segmentation](https://mlsanity.com/paper/2004.04534/) (2020)
- [CAVER: Cross-Modal View-Mixed Transformer for Bi-Modal Salient Object Detection](https://mlsanity.com/paper/2112.02363/) (2021)
- [ISSAFE: Improving Semantic Segmentation in Accidents by Fusing Event-based Data](https://mlsanity.com/paper/2008.08974/) (2020)
- [MMANet: Margin-aware Distillation and Modality-aware Regularization for Incomplete Multimodal Learning](https://mlsanity.com/paper/2304.08028/) (2023)
- [CIR-Net: Cross-modality Interaction and Refinement for RGB-D Salient Object Detection](https://mlsanity.com/paper/2210.02843/) (2022)
- [X-Align: Cross-Modal Cross-View Alignment for Bird's-Eye-View Segmentation](https://mlsanity.com/paper/2210.06778/) (2022)
- [Semantics, Distortion, and Style Matter: Towards Source-free UDA for Panoramic Segmentation](https://mlsanity.com/paper/2403.12505/) (2024)
- [Masked Jigsaw Puzzle: A Versatile Position Embedding for Vision Transformers](https://mlsanity.com/paper/2205.12551/) (2022)
- [ExACT: Language-guided Conceptual Reasoning and Uncertainty Estimation for Event-based Action Recognition and More](https://mlsanity.com/paper/2403.12534/) (2024)
- [MUSES: The Multi-Sensor Semantic Perception Dataset for Driving under Uncertainty](https://mlsanity.com/paper/2401.12761/) (2024)
- [HRFuser: A Multi-resolution Sensor Fusion Architecture for 2D Object Detection](https://mlsanity.com/paper/2206.15157/) (2022)
- [Learning Modality-agnostic Representation for Semantic Segmentation from Any Modalities](https://mlsanity.com/paper/2407.11351/) (2024)

## Built on by (most cited first, 9 of 9)

- [MLLMs are Deeply Affected by Modality Bias](https://mlsanity.com/paper/2505.18657/) (2025)
- [Multimodal Spatial Reasoning in the Large Model Era: A Survey and Benchmarks](https://mlsanity.com/paper/2510.25760/) (2025)
- [EGFormer: Towards Efficient and Generalizable Multimodal Semantic Segmentation](https://mlsanity.com/paper/2505.14014/) (2025)
- [BiXFormer: A Robust Framework for Maximizing Modality Effectiveness in Multi-Modal Semantic Segmentation](https://mlsanity.com/paper/2506.03675/) (2025)
- [Partial CLIP is Enough: Chimera-Seg for Zero-shot Semantic Segmentation](https://mlsanity.com/paper/2506.22032/) (2025)
- [TIRAuxCloud: A Thermal Infrared Dataset for Day and Night Cloud Detection](https://mlsanity.com/paper/2602.21905/) (2026)
- [Parameter-Efficient Modality-Balanced Symmetric Fusion for Multimodal Remote Sensing Semantic Segmentation](https://mlsanity.com/paper/2603.17705/) (2026)
- [UMSS: Towards Unsupervised Multi-modal Semantic Segmentation](https://mlsanity.com/paper/2607.12372/) (2026)
- [GraphWrit3R: End-to-End 3D Scene Graph Writing](https://mlsanity.com/paper/2609.31595/) (2026)

## Abstract (arXiv)

Fusing and balancing multi-modal inputs from novel sensors for dense prediction tasks, particularly semantic segmentation, is critically important yet remains a significant challenge. One major limitation is the tendency of multi-modal frameworks to over-rely on easily learnable modalities, a phenomenon referred to as unimodal dominance or bias. This issue becomes especially problematic in real-world scenarios where the dominant modality may be unavailable, resulting in severe performance degradation. To this end, we apply a simple but effective plug-and-play regularization term based on functional entropy, which introduces no additional parameters or modules. This term is designed to intuitively balance the contribution of each visual modality to the segmentation results. Specifically, we leverage the log-Sobolev inequality to bound functional entropy using functional-Fisher-information. By maximizing the information contributed by each visual modality, our approach mitigates unimodal dominance and establishes a more balanced and robust segmentation framework. A multi-scale regularization module is proposed to apply our proposed plug-and-play term on high-level features and also segmentation predictions for more balanced multi-modal learning. Extensive experiments on three datasets demonstrate that our proposed method achieves superior performance, i.e., +13.94%, +3.25%, and +3.64%, without introducing any additional parameters.