mlsanityData to 14 Oct 2026

Residual Optimal Transport-Based Experts Collaboration Towards Modality-Aware Infrared-Visible Object Detection

arXiv 2609.03516 ↗First posted 3 Sep 2026cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Infrared-visible object detection (IVOD) integrates complementary evidence from visible and infrared sensors for reliable perception in challenging scenes. In practice, sensors may fail or drop frames, leaving one modality unavailable or intermittent. Existing methods for IVOD assume both modalities are always present, and fixed fusion collapses when one stream is missing. Furthermore, it remains a critical challenge to reliably estimate semantic correlation across heterogeneous modalities, especially under spectral distribution discrepancy. We present FlexibleFusion, a unified and adaptive method that flexibly allocates integration pathways and fusion strength, operating seamlessly across complete and missing-modality regimes. At its core, the Modality-Aware Experts Collaboration (MAEC) mechanism selectively activates and aggregates cross-modal or intra-modal expert pathways. It allows cross-modal fusion when full modalities are available and falls back to self-fusion under missing conditions. Additionally, we design Residual Self-Paced Entropic Optimal Transport (RSPEOT) to align heterogeneous feature distributions from a transport perspective. Instead of relying on the fixed sparsity coefficient in standard entropic optimal transport (EOT), RSPEOT introduces a residual-driven self-paced update that prioritizes reliable matches and progressively refines harder ones. This design alleviates the additional optimization burden of standard EOT while preserving reliable semantic alignment. Comprehensive experiments under complete and missing-modality protocols show consistent performance across arbitrary modality configurations. Code will be released upon publication.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 27 of its 56 references that are in the map. First posted on arXiv 3 Sep 2026.

Builds on

27 of 56 references.

References · 27

  1. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  2. 2016Feature Pyramid Networks for Object Detectioncited by 5,036
  3. 2013Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distancescited by 2,316
  4. 2022DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detectioncited by 1,319
  5. 2022Target-aware Dual Adversarial Learning and a Multi-scenario Multi-Modality Benchmark to Fuse Infrared and Visible for Object Detectioncited by 317
  6. 2021SMIL: Multimodal Learning with Severely Missing Modalitycited by 245
  7. 2021LLVIP: A Visible-infrared Paired Dataset for Low-light Visioncited by 231
  8. 2020Drone-based RGB-Infrared Cross-Modality Vehicle Detection via Uncertainty-Aware Learningcited by 152
  9. 2022Neural Optimal Transportcited by 126
  10. 2019Missing MRI Pulse Sequence Synthesis using Multi-Modal Generative Adversarial Networkcited by 98
  11. 2024Deep Multimodal Learning with Missing Modality: A Surveycited by 92
  12. 2021Cross-Modality Fusion Transformer for Multispectral Object Detectioncited by 86
15 more
  1. 2020Multispectral Fusion for Object Detection with Cyclic Fuse-and-Refine Blockscited by 85
  2. 2023ICAFusion: Iterative Cross-Attention Guided Feature Fusion for Multispectral Object Detectioncited by 76
  3. 2021Multimodal Object Detection via Probabilistic Ensemblingcited by 68
  4. 2024Fusion-Mamba for Cross-modality Object Detectioncited by 67
  5. 2023C2Former: Calibrated and Complementary Transformer for RGB-Infrared Object Detectioncited by 58
  6. 2022Towards Good Practices for Missing Modality Robust Action Recognitioncited by 54
  7. 2023Unified Multi-Modal Image Synthesis for Missing Modality Imputationcited by 40
  8. 2023Robust Multimodal Learning with Missing Modalities via Parameter-Efficient Adaptationcited by 29
  9. 2024DAMSDet: Dynamic Adaptive Multispectral Detection Transformer with Competitive Query Selection and Adaptive Feature Fusioncited by 28
  10. 2023TFDet: Target-Aware Fusion for RGB-T Pedestrian Detectioncited by 18
  11. 2016Cross-Modal Scene Networkscited by 16
  12. 2019A unified representation network for segmentation with missing modalitiescited by 15
  13. 2024Gradient-Guided Modality Decoupling for Missing-Modality Robustnesscited by 11
  14. 2024FD2-Net: Frequency-Driven Feature Decomposition Network for Infrared-Visible Object Detectioncited by 10
  15. 2025On Modality Incomplete Infrared-Visible Object Detection: An Architecture Compatibility Perspectivecited by 1

Built on by

Not cited yet.