Bridging Multimodal Fusion and Expert Routing via Spectral Reliability Descriptors for Robust Object Detection
On the map: Mixture-of-experts → Vision mixture-of-experts
Abstract (arXiv)
RGB-infrared detectors typically discard the statistics generated during cross-modal fusion, leaving downstream modules unaware of whether the current interaction is reliable. We propose to extract a parameter-free, 7-dimensional spectral reliability descriptor -- summarizing band energy, amplitude ratio, phase consistency, and cross-modal correlation -- and to reuse it beyond the fusion stage. The descriptor drives both Spectral Reliability Fusion (SRF), which gates a spectral residual against a conservative spatial base, and Reliability-Conditioned Expert Routing (RCER), which combines the descriptor with pooled content to steer sparse post-fusion experts. Under matched ablations, descriptor-aware gating improves mAP50 over content-only adaptive gating; a 2×2 factorial analysis further shows that descriptor-conditioned routing provides the larger marginal gain over expert architecture alone at near-equal parameter count. Under six synthetic degradations on DroneVehicle, average retention rises to 95.0%, versus 92.0% for content-only MoE and 87.9% for concatenation, with the largest gain under modality drop; the same model also improves mAP50 by +5.2/+5.3 on the natural day/night split. These results suggest that preserving fusion-time reliability as an explicit signal benefits both adaptive fusion and post-fusion conditional computation.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Vision mixture-of-experts, one of 12 directions in Mixture-of-experts. Builds on 18 of its 35 references that are in the map. First posted on arXiv 31 May 2026.
Builds on
18 of 35 references.
References · 18
- 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
- 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,352
- 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 864
- 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 399
- 2022Target-aware Dual Adversarial Learning and a Multi-scenario Multi-Modality Benchmark to Fuse Infrared and Visible for Object Detectioncited by 317
- 2021Global Filter Networks for Image Classificationcited by 260
- 2020VarifocalNet: An IoU-aware Dense Object Detectorcited by 231
- 2022CDDFuse: Correlation-Driven Dual-Branch Feature Decomposition for Multi-Modality Image Fusioncited by 223
- 2021Dynamic Head: Unifying Object Detection Heads with Attentionscited by 187
- 2020Drone-based RGB-Infrared Cross-Modality Vehicle Detection via Uncertainty-Aware Learningcited by 152
- 2024D-FINE: Redefine Regression Task in DETRs as Fine-grained Distribution Refinementcited by 74
- 2024DEIM: DETR with Improved Matching for Fast Convergencecited by 68
6 more
- 2023Learning a Graph Neural Network with Cross Modality Interaction for Image Fusioncited by 28
- 2024Missing Modality Prediction for Unpaired Multimodal Learning via Joint Embedding of Unimodal Modelscited by 20
- 2024Removal then Selection: A Coarse-to-Fine Fusion Perspective for RGB-Infrared Object Detectioncited by 16
- 2024TSJNet: A Multi-modality Target and Semantic Awareness Joint-driven Image Fusion Networkcited by 10
- 2025FreDFT: Frequency Domain Fusion Transformer for Visible-Infrared Object Detectioncited by 3
- 2025AlignFreeNet: Is Cross-Modal Pre-Alignment Necessary? An End-to-End Alignment-Free Lightweight Network for Visible-Infrared Object Detectioncited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026LER-YOLO: Reliability-Aware Expert Routing for Misaligned RGB-Infrared UAV Detection
- 2024Cocoon: Robust Multi-Modal Perception with Uncertainty-Aware Sensor Fusion
- 2025Multispectral Detection Transformer with Infrared-Centric Feature Fusion
- 2026Variance-Guided Spatial Attention Fusion for Robust End-to-End Driving under Asymmetric Sensor Degradation
- 2026SpectraDINO: Modality-Conditioned Adaptation of RGB Vision Foundation Models Across Infrared Bands
- 2023Uncertainty-Encoded Multi-Modal Fusion for Robust Object Detection in Autonomous Driving
- 2024Rethinking Early-Fusion Strategies for Improved Multispectral Object Detection
- 2026FreqKD: Frequency-Decoupled Cross-Modal Knowledge Distillation for Infrared Object Detection