mlsanityData to 7 Oct 2026

E2E-MFD: Towards End-to-End Synchronous Multimodal Fusion Detection

arXiv 2403.09323 ↗First posted 14 Mar 2024cs.CVcited by 21

Not on the map. Why?

Abstract (arXiv)

Multimodal image fusion and object detection are crucial for autonomous driving. While current methods have advanced the fusion of texture details and semantic information, their complex training processes hinder broader applications. Addressing this challenge, we introduce E2E-MFD, a novel end-to-end algorithm for multimodal fusion detection. E2E-MFD streamlines the process, achieving high performance with a single training phase. It employs synchronous joint optimization across components to avoid suboptimal solutions tied to individual tasks. Furthermore, it implements a comprehensive optimization strategy in the gradient matrix for shared parameters, ensuring convergence to an optimal fusion detection configuration. Our extensive testing on multiple public datasets reveals E2E-MFD's superior capabilities, showcasing not only visually appealing image fusion but also impressive detection outcomes, such as a 3.9% and 2.0% mAP50 increase on horizontal object detection dataset M3FD and oriented object detection dataset DroneVehicle, respectively, compared to state-of-the-art approaches. The code is released at https://github.com/icey-zhang/E2E-MFD.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Builds on 41 of its 70 references that are in the map. First posted on arXiv 14 Mar 2024.

Builds on

41 of 70 references.

References · 41

  1. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  2. 2019MMDetection: Open MMLab Detection Toolbox and Benchmarkcited by 1,379
  3. 2017Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semanticscited by 1,345
  4. 2020Gradient Surgery for Multi-Task Learningcited by 1,210
  5. 2017GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networkscited by 816
  6. 2017An Overview of Multi-Task Learning in Deep Neural Networkscited by 791
  7. 2021Conflict-Averse Gradient Descent for Multi-task Learningcited by 441
  8. 2022DiffusionDet: Diffusion Model for Object Detectioncited by 354
  9. 2022Target-aware Dual Adversarial Learning and a Multi-scenario Multi-Modality Benchmark to Fuse Infrared and Visible for Object Detectioncited by 313
  10. 2023A Survey on Multimodal Large Language Models for Autonomous Drivingcited by 288
  11. 2022CDDFuse: Correlation-Driven Dual-Branch Feature Decomposition for Multi-Modality Image Fusioncited by 222
  12. 2022Multi-Task Learning as a Bargaining Gamecited by 209
29 more
  1. 2016UberNet: Training a `Universal' Convolutional Neural Network for Low-, Mid-, and High-Level Vision using Diverse Datasets and Limited Memorycited by 200
  2. 2019R3Det: Refined Single-Stage Detector with Feature Refinement for Rotating Objectcited by 193
  3. 2019AdaShare: Learning What To Share For Efficient Deep Multi-Task Learningcited by 159
  4. 2023DDFM: Denoising Diffusion Model for Multi-Modality Image Fusioncited by 153
  5. 2020Drone-based RGB-Infrared Cross-Modality Vehicle Detection via Uncertainty-Aware Learningcited by 150
  6. 2019Gliding vertex on the horizontal bounding box for multi-oriented object detectioncited by 134
  7. 2023Multi-interactive Feature Learning and a Full-time Multi-modality Benchmark for Image Fusion and Segmentationcited by 128
  8. 2017Routing Networks: Adaptive Selection of Non-linear Functions for Multi-Task Learningcited by 125
  9. 2022MMRotate: A Rotated Object Detection Benchmark using PyTorchcited by 115
  10. 2021RFN-Nest: An end-to-end residual fusion network for infrared and visible imagescited by 109
  11. 2023Large Selective Kernel Network for Remote Sensing Object Detectioncited by 98
  12. 2015Multi-task Sequence to Sequence Learningcited by 94
  13. 2023LRRNet: A Novel Representation Learning Guided Fusion Network for Infrared and Visible Imagescited by 87
  14. 2021Cross-Modality Fusion Transformer for Multispectral Object Detectioncited by 86
  15. 2022Unsupervised Misaligned Infrared and Visible Image Fusion via Cross-Modality Image Generation and Registrationcited by 80
  16. 2019Branched Multi-Task Networks: Deciding What Layers To Sharecited by 76
  17. 2023ICAFusion: Iterative Cross-Attention Guided Feature Fusion for Multispectral Object Detectioncited by 76
  18. 2020DIDFuse: Deep Image Decomposition for Infrared and Visible Image Fusioncited by 72
  19. 2023Independent Component Alignment for Multi-Task Learningcited by 69
  20. 2020SCRDet++: Detecting Small, Cluttered and Rotated Objects via Instance-Level Feature Denoising and Rotation Loss Smoothingcited by 62
  21. 2020NestFuse: An Infrared and Visible Image Fusion Architecture based on Nest Connection and Spatial/Channel Attention Modelscited by 62
  22. 2022Translation, Scale and Rotation: Cross-Modal Alignment Meets RGB-Infrared Vehicle Detectioncited by 49
  23. 2021Edge-aware Guidance Fusion Network for RGB Thermal Scene Parsingcited by 44
  24. 2019Stochastic Filter Groups for Multi-Task CNNs: Learning Specialist and Generalist Convolution Kernelscited by 42
  25. 2022Detecting Rotated Objects as Gaussian Distributions and Its 3-D Generalizationcited by 36
  26. 2023Bi-level Dynamic Learning for Jointly Multi-modality Image Fusion and Beyondcited by 26
  27. 2019Deep Elastic Networks with Model Selection for Multi-Task Learningcited by 20
  28. 2024DA-BEV: Unsupervised Domain Adaptation for Bird's Eye View Perceptioncited by 5
  29. 2024Domain Adaptation for Large-Vocabulary Object Detectorscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 15; 2026: 4.

21 papers.

Sort:

By area:

  1. 2024MaeFuse: Transferring Omni Features with Pretrained Masked Autoencoders for Infrared and Visible Image Fusion via Guided Trainingcited by 15
  2. 2025TurboTrain: Towards Efficient and Balanced Multi-Task Learning for Multi-Agent Perception and Predictioncited by 9
  3. 2025When Continue Learning Meets Multimodal Large Language Model: A Surveycited by 8
  4. 2025M4-SAR: A Multi-Resolution, Multi-Polarization, Multi-Scene, Multi-Source Dataset and Benchmark for optical-SAR Object Detectioncited by 8
  5. 2025Seeing Through the Noise: Improving Infrared Small Target Detection and Segmentation from Noise Suppression Perspectivecited by 7
  6. 2025A Survey of Multi-sensor Fusion Perception for Embodied AI: Background, Methods, Challenges and Prospectscited by 5
  7. 2025Towards Unified Semantic and Controllable Image Fusion: A Diffusion Transformer Approachcited by 4
  8. 2025Cross-modal Offset-guided Dynamic Alignment and Fusion for Weakly Aligned UAV Object Detectioncited by 3
  9. 2024FusionSAM: Visual Multi-Modal Learning with Segment Anythingcited by 1
  10. 2025DOD-SA: Infrared-Visible Decoupled Object Detection with Single-Modality Annotationscited by 1
  11. 2026Physical Adversarial Clothing Evades Visible-Thermal Detectors via Non-Overlapping RGB-T Patterncited by 1
  12. 2025RGBX-DiffusionDet: A Framework for Multi-Modal RGB-X Object Detection Using DiffusionDetcited by 0
9 more
  1. 2025MultiTaskVIF: Segmentation-oriented visible and infrared image fusion via multi-task learningcited by 0
  2. 2025LSFDNet: A Single-Stage Fusion and Detection Network for Ships Using SWIR and LWIRcited by 0
  3. 2025HiddenObject: Modality-Agnostic Fusion for Multimodal Hidden Object Detectioncited by 0
  4. 2025RIS-FUSION: Rethinking Text-Driven Infrared and Visible Image Fusion from the Perspective of Referring Image Segmentationcited by 0
  5. 2025MO R-CNN: Multispectral Oriented R-CNN for Object Detection in Remote Sensing Imagecited by 0
  6. 2025SSVIF: Self-Supervised Segmentation-Oriented Visible and Infrared Image Fusioncited by 0
  7. 2026LDFE: Laplacian Decoupled Feature Enhancement Block for Dual-Stream CNN-based RGB-IR Object Detectioncited by 0
  8. 2026EGM-Det: Entropy-Guided Multimodal Adaptive Fusion for UAV RGB-IR Object Detectioncited by 0
  9. 2026IT-TextFusion: Iterative Text-Image Interaction with Text-Guided Residual Refinement for Degradation-Aware Image Fusioncited by 0