# A Multi-modal Fusion Network for Terrain Perception Based on Illumination Aware

arXiv: https://arxiv.org/abs/2505.11066 · HTML: https://mlsanity.com/paper/2505.11066/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 37 references that are in the map.
- First posted on arXiv 16 May 2025.

## Builds on (most cited first, 12 of 12)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [SurroundOcc: Multi-Camera 3D Occupancy Prediction for Autonomous Driving](https://mlsanity.com/paper/2303.09551/) (2023)
- [Multimodal Token Fusion for Vision Transformers](https://mlsanity.com/paper/2204.08721/) (2022)
- [MMTM: Multimodal Transfer Module for CNN Fusion](https://mlsanity.com/paper/1911.08670/) (2019)
- [Deep Multimodal Fusion by Channel Exchanging](https://mlsanity.com/paper/2011.05005/) (2020)
- [Multimodal Fusion Transformer for Remote Sensing Image Classification](https://mlsanity.com/paper/2203.16952/) (2022)
- [Multi-Modal Data-Efficient 3D Scene Understanding for Autonomous Driving](https://mlsanity.com/paper/2405.05258/) (2024)
- [Self-Supervised Visual Terrain Classification from Unsupervised Acoustic Feature Learning](https://mlsanity.com/paper/1912.03227/) (2019)
- [MMSFormer: Multimodal Transformer for Material and Semantic Segmentation](https://mlsanity.com/paper/2309.04001/) (2023)
- [Grasp State Assessment of Deformable Objects Using Visual-Tactile Fusion Perception](https://mlsanity.com/paper/2006.12729/) (2020)
- [Elastic Tactile Simulation Towards Tactile-Visual Perception](https://mlsanity.com/paper/2108.05013/) (2021)

## Built on by (most cited first, 1 of 1)

- [Synesthesia of Vehicles: Tactile Data Synthesis from Visual Inputs](https://mlsanity.com/paper/2602.01832/) (2026)

## Abstract (arXiv)

Road terrains play a crucial role in ensuring the driving safety of autonomous vehicles (AVs). However, existing sensors of AVs, including cameras and Lidars, are susceptible to variations in lighting and weather conditions, making it challenging to achieve real-time perception of road conditions. In this paper, we propose an illumination-aware multi-modal fusion network (IMF), which leverages both exteroceptive and proprioceptive perception and optimizes the fusion process based on illumination features. We introduce an illumination-perception sub-network to accurately estimate illumination features. Moreover, we design a multi-modal fusion network which is able to dynamically adjust weights of different modalities according to illumination features. We enhance the optimization process by pre-training of the illumination-perception sub-network and incorporating illumination loss as one of the training constraints. Extensive experiments demonstrate that the IMF shows a superior performance compared to state-of-the-art methods. The comparison results with single modality perception methods highlight the comprehensive advantages of multi-modal fusion in accurately perceiving road terrains under varying lighting conditions. Our dataset is available at: https://github.com/lindawang2016/IMF.