Layer-Wise Modality Decomposition for Interpretable Multimodal Sensor Fusion
Not on the map. Why?
Abstract (arXiv)
In autonomous driving, transparency in the decision-making of perception models is critical, as even a single misperception can be catastrophic. Yet with multi-sensor inputs, it is difficult to determine how each modality contributes to a prediction because sensor information becomes entangled within the fusion network. We introduce Layer-Wise Modality Decomposition (LMD), a post-hoc, model-agnostic interpretability method that disentangles modality-specific information across all layers of a pretrained fusion model. To our knowledge, LMD is the first approach to attribute the predictions of a perception model to individual input modalities in a sensor-fusion system for autonomous driving. We evaluate LMD on pretrained fusion models under camera-radar, camera-LiDAR, and camera-radar-LiDAR settings for autonomous driving. Its effectiveness is validated using structured perturbation-based metrics and modality-wise visual decompositions, demonstrating practical applicability to interpreting high-capacity multimodal architectures. Code is available at https://github.com/detxter-jvb/Layer-Wise-Modality-Decomposition.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 29 of its 40 references that are in the map. First posted on arXiv 2 Nov 2025.
Builds on
29 of 40 references.
References · 29
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
- 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
- 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
- 2017A Unified Approach to Interpreting Model Predictionscited by 6,555
- 2016Layer Normalizationcited by 5,669
- 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
- 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,422
- 2017Grad-CAM++: Improved Visual Explanations for Deep Convolutional Networkscited by 939
- 2022BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representationcited by 850
- 2016Training Deep Nets with Sublinear Memory Costcited by 848
- 2020Transformer Interpretability Beyond Attention Visualizationcited by 627
17 more
- 2019Score-CAM: Score-Weighted Visual Explanations for Convolutional Neural Networkscited by 582
- 2020Neural Additive Models: Interpretable Machine Learning with Neural Netscited by 319
- 2015Explaining NonLinear Classification Decisions with Deep Taylor Decompositioncited by 250
- 2016Layer-wise Relevance Propagation for Neural Networks with Local Renormalization Layerscited by 199
- 2018Local Rule-Based Explanations of Black Box Decision Systemscited by 166
- 2022Simple-BEV: What Really Matters for Multi-Sensor BEV Perception?cited by 128
- 2024AttnLRP: Attention-Aware Layer-Wise Relevance Propagation for Transformerscited by 120
- 2022XAI for Transformers: Better Explanations through Conservative Propagationcited by 100
- 2021NODE-GAM: Neural Generalized Additive Model for Interpretable Deep Learningcited by 88
- 2023CRN: Camera Radar Net for Accurate, Robust, Efficient 3D Perceptioncited by 82
- 2024RCBEVDet: Radar-camera Fusion in Bird's Eye View for 3D Object Detectioncited by 82
- 2020Analyzing the Source and Target Contributions to Predictions in Neural Machine Translationcited by 71
- 2022Neural Basis Models for Interpretabilitycited by 56
- 2020Towards Interpretable Semantic Segmentation via Gradient-weighted Class Activation Mappingcited by 49
- 2019Purifying Interaction Effects with the Functional ANOVA: An Efficient Algorithm for Recovering Identifiable Additive Modelscited by 42
- 2024Gaussian Process Neural Additive Modelscited by 11
- 2024Layer-Wise Relevance Propagation with Conservation Property for ResNetcited by 7
Built on by
Not cited yet.