mlsanityData to 14 Oct 2026

Bridging Multimodal Fusion and Expert Routing via Spectral Reliability Descriptors for Robust Object Detection

arXiv 2606.01173 ↗First posted 31 May 2026cs.CVcited by 0

On the map: Mixture-of-experts → Vision mixture-of-experts

Abstract (arXiv)

RGB-infrared detectors typically discard the statistics generated during cross-modal fusion, leaving downstream modules unaware of whether the current interaction is reliable. We propose to extract a parameter-free, 7-dimensional spectral reliability descriptor -- summarizing band energy, amplitude ratio, phase consistency, and cross-modal correlation -- and to reuse it beyond the fusion stage. The descriptor drives both Spectral Reliability Fusion (SRF), which gates a spectral residual against a conservative spatial base, and Reliability-Conditioned Expert Routing (RCER), which combines the descriptor with pooled content to steer sparse post-fusion experts. Under matched ablations, descriptor-aware gating improves mAP50 over content-only adaptive gating; a 2×2 factorial analysis further shows that descriptor-conditioned routing provides the larger marginal gain over expert architecture alone at near-equal parameter count. Under six synthetic degradations on DroneVehicle, average retention rises to 95.0%, versus 92.0% for content-only MoE and 87.9% for concatenation, with the largest gain under modality drop; the same model also improves mAP50 by +5.2/+5.3 on the natural day/night split. These results suggest that preserving fusion-time reliability as an explicit signal benefits both adaptive fusion and post-fusion conditional computation.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Vision mixture-of-experts, one of 12 directions in Mixture-of-experts. Builds on 18 of its 35 references that are in the map. First posted on arXiv 31 May 2026.

Builds on

18 of 35 references.

References · 18

  1. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  2. 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,352
  3. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 864
  4. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 399
  5. 2022Target-aware Dual Adversarial Learning and a Multi-scenario Multi-Modality Benchmark to Fuse Infrared and Visible for Object Detectioncited by 317
  6. 2021Global Filter Networks for Image Classificationcited by 260
  7. 2020VarifocalNet: An IoU-aware Dense Object Detectorcited by 231
  8. 2022CDDFuse: Correlation-Driven Dual-Branch Feature Decomposition for Multi-Modality Image Fusioncited by 223
  9. 2021Dynamic Head: Unifying Object Detection Heads with Attentionscited by 187
  10. 2020Drone-based RGB-Infrared Cross-Modality Vehicle Detection via Uncertainty-Aware Learningcited by 152
  11. 2024D-FINE: Redefine Regression Task in DETRs as Fine-grained Distribution Refinementcited by 74
  12. 2024DEIM: DETR with Improved Matching for Fast Convergencecited by 68
6 more
  1. 2023Learning a Graph Neural Network with Cross Modality Interaction for Image Fusioncited by 28
  2. 2024Missing Modality Prediction for Unpaired Multimodal Learning via Joint Embedding of Unimodal Modelscited by 20
  3. 2024Removal then Selection: A Coarse-to-Fine Fusion Perspective for RGB-Infrared Object Detectioncited by 16
  4. 2024TSJNet: A Multi-modality Target and Semantic Awareness Joint-driven Image Fusion Networkcited by 10
  5. 2025FreDFT: Frequency Domain Fusion Transformer for Visible-Infrared Object Detectioncited by 3
  6. 2025AlignFreeNet: Is Cross-Modal Pre-Alignment Necessary? An End-to-End Alignment-Free Lightweight Network for Visible-Infrared Object Detectioncited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026LER-YOLO: Reliability-Aware Expert Routing for Misaligned RGB-Infrared UAV Detection
  2. 2024Cocoon: Robust Multi-Modal Perception with Uncertainty-Aware Sensor Fusion
  3. 2025Multispectral Detection Transformer with Infrared-Centric Feature Fusion
  4. 2026Variance-Guided Spatial Attention Fusion for Robust End-to-End Driving under Asymmetric Sensor Degradation
  5. 2026SpectraDINO: Modality-Conditioned Adaptation of RGB Vision Foundation Models Across Infrared Bands
  6. 2023Uncertainty-Encoded Multi-Modal Fusion for Robust Object Detection in Autonomous Driving
  7. 2024Rethinking Early-Fusion Strategies for Improved Multispectral Object Detection
  8. 2026FreqKD: Frequency-Decoupled Cross-Modal Knowledge Distillation for Infrared Object Detection