# Revisiting Modality Imbalance In Multimodal Pedestrian Detection

arXiv: https://arxiv.org/abs/2302.12589 · HTML: https://mlsanity.com/paper/2302.12589/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 24 references that are in the map.
- First posted on arXiv 24 Feb 2023.

## Builds on (most cited first, 12 of 12)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [Improving Multispectral Pedestrian Detection by Addressing Modality Imbalance Problems](https://mlsanity.com/paper/2008.03043/) (2020)
- [Multispectral Deep Neural Networks for Pedestrian Detection](https://mlsanity.com/paper/1611.02644/) (2016)
- [Repulsion Loss: Detecting Pedestrians in a Crowd](https://mlsanity.com/paper/1711.07752/) (2017)
- [Multispectral Pedestrian Detection via Simultaneous Detection and Segmentation](https://mlsanity.com/paper/1808.04818/) (2018)
- [RGB and LiDAR fusion based 3D Semantic Segmentation for Autonomous Driving](https://mlsanity.com/paper/1906.00208/) (2019)
- [Spatio-Contextual Deep Network Based Multimodal Pedestrian Detection For Autonomous Driving](https://mlsanity.com/paper/2105.12713/) (2021)
- [TiledSoilingNet: Tile-level Soiling Detection on Automotive Surround-view Cameras Using Coverage Metric](https://mlsanity.com/paper/2007.00801/) (2020)
- [SoildNet: Soiling Degradation Detection in Autonomous Driving](https://mlsanity.com/paper/1911.01054/) (2019)
- [Deep Multi-Task Networks For Occluded Pedestrian Pose Estimation](https://mlsanity.com/paper/2206.07510/) (2022)

## Built on by (most cited first, 10 of 10)

- [DynCIM: Dynamic Curriculum for Imbalanced Multimodal Learning](https://mlsanity.com/paper/2503.06456/) (2025)
- [Pedestrian Detection in Low-Light Conditions: A Comprehensive Survey](https://mlsanity.com/paper/2401.07801/) (2024)
- [MiPa: Mixed Patch Infrared-Visible Modality Agnostic Object Detection](https://mlsanity.com/paper/2404.18849/) (2024)
- [BEVSeg2TP: Surround View Camera Bird's-Eye-View Based Joint Vehicle Segmentation and Ego Vehicle Trajectory Prediction](https://mlsanity.com/paper/2312.13081/) (2023)
- [MSCoTDet: Language-driven Multi-modal Fusion for Improved Multispectral Pedestrian Detection](https://mlsanity.com/paper/2403.15209/) (2024)
- [Fisheye Camera and Ultrasonic Sensor Fusion For Near-Field Obstacle Perception in Bird's-Eye-View](https://mlsanity.com/paper/2402.00637/) (2024)
- [Reflective Teacher: Semi-Supervised Multimodal 3D Object Detection in Bird's-Eye-View via Uncertainty Measure](https://mlsanity.com/paper/2412.04337/) (2024)
- [Enhance-then-Balance Modality Collaboration for Robust Multimodal Sentiment Analysis](https://mlsanity.com/paper/2604.12518/) (2026)
- [BEVMOSNet: Multimodal Fusion for BEV Moving Object Segmentation](https://mlsanity.com/paper/2503.03280/) (2025)
- [Confidence-driven Gradient Modulation for Multimodal Human Activity Recognition: A Dynamic Contrastive Dual-Path Learning Approach](https://mlsanity.com/paper/2507.02826/) (2025)

## Abstract (arXiv)

Multimodal learning, particularly for pedestrian detection, has recently received emphasis due to its capability to function equally well in several critical autonomous driving scenarios such as low-light, night-time, and adverse weather conditions. However, in most cases, the training distribution largely emphasizes the contribution of one specific input that makes the network biased towards one modality. Hence, the generalization of such models becomes a significant problem where the non-dominant input modality during training could be contributing more to the course of inference. Here, we introduce a novel training setup with regularizer in the multimodal architecture to resolve the problem of this disparity between the modalities. Specifically, our regularizer term helps to make the feature fusion method more robust by considering both the feature extractors equivalently important during the training to extract the multimodal distribution which is referred to as removing the imbalance problem. Furthermore, our decoupling concept of output stream helps the detection task by sharing the spatial sensitive information mutually. Extensive experiments of the proposed method on KAIST and UTokyo datasets shows improvement of the respective state-of-the-art performance.