mlsanityData to 7 Oct 2026

Dynamic Multimodal Fusion

arXiv 2204.00102 ↗First posted 31 Mar 2022cs.CV cs.AI cs.MMcited by 54

Not on the map. Why?

Abstract (arXiv)

Deep multimodal learning has achieved great progress in recent years. However, current fusion approaches are static in nature, i.e., they process and fuse multimodal inputs with identical computation, without accounting for diverse computational demands of different multimodal data. In this work, we propose dynamic multimodal fusion (DynMM), a new approach that adaptively fuses multimodal data and generates data-dependent forward paths during inference. To this end, we propose a gating function to provide modality-level or fusion-level decisions on-the-fly based on multimodal features and a resource-aware loss function that encourages computational efficiency. Results on various multimodal tasks demonstrate the efficiency and wide applicability of our approach. For instance, DynMM can reduce the computation costs by 46.5% with only a negligible accuracy loss (CMU-MOSEI sentiment analysis) and improve segmentation performance with over 21% savings in computation (NYU Depth V2 semantic segmentation) when compared with static fusion approaches. We believe our approach opens a new direction towards dynamic multimodal network design, with applications to a wide range of multimodal tasks.

Key facts. Cited by 54 papers in the map (data through 7 Oct 2026). Builds on 40 of its 55 references that are in the map. First posted on arXiv 31 Mar 2022.

Builds on

40 of 55 references.

References · 40

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2017Squeeze-and-Excitation Networkscited by 4,763
  3. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  4. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  5. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,004
  6. 2017Dynamic Routing Between Capsulescited by 689
  7. 2017BranchyNet: Fast Inference via Early Exiting from Deep Neural Networkscited by 591
  8. 2016Adaptive Computation Time for Recurrent Neural Networkscited by 479
  9. 2017Tensor Fusion Network for Multimodal Sentiment Analysiscited by 429
  10. 2021Dynamic Neural Networks: A Surveycited by 405
  11. 2021Attention Bottlenecks for Multimodal Fusioncited by 361
  12. 2017SkipNet: Learning Dynamic Routing in Convolutional Networkscited by 328
28 more
  1. 2019CondConv: Conditionally Parameterized Convolutions for Efficient Inferencecited by 273
  2. 2018Efficient Low-rank Multimodal Fusion with Modality-Specific Factorscited by 268
  3. 2017Gated Multimodal Units for Information Fusioncited by 244
  4. 2015Conditional Computation in Neural Networks for faster modelscited by 166
  5. 2019Listen to Look: Action Recognition by Previewing Audiocited by 163
  6. 2017Adaptive Neural Networks for Efficient Inferencecited by 161
  7. 2021MultiBench: Multiscale Benchmarks for Multimodal Representation Learningcited by 143
  8. 2019MMTM: Multimodal Transfer Module for CNN Fusioncited by 128
  9. 2017CondenseNet: An Efficient DenseNet using Learned Group Convolutionscited by 121
  10. 2020Bi-directional Cross-Modality Feature Propagation with Separation-and-Aggregation Gate for RGB-D Semantic Segmentationcited by 120
  11. 2020Deep Multimodal Fusion by Channel Exchangingcited by 117
  12. 2019ACNet: Attention Based Network to Exploit Complementary Features for RGBD Semantic Segmentationcited by 105
  13. 2020Glance and Focus: a Dynamic Approach to Reducing Spatial Redundancy in Image Classificationcited by 99
  14. 2018CentralNet: a Multilayer Approach for Multimodal Fusioncited by 86
  15. 2019MFAS: Multimodal Fusion Architecture Searchcited by 84
  16. 2020Efficient RGB-D Semantic Segmentation for Indoor Scene Analysiscited by 83
  17. 2020Learning Dynamic Routing for Semantic Segmentationcited by 79
  18. 2017Attention-Based Multimodal Fusion for Video Descriptioncited by 75
  19. 2021Deep RGB-D Saliency Detection with Depth-Sensitive Attention and Automatic Multi-Modal Fusioncited by 56
  20. 2019Adaptive Weighting Multi-Field-of-View CNN for Semantic Segmentation in Pathologycited by 36
  21. 2021AdaMML: Adaptive Multi-Modal Learning for Efficient Video Recognitioncited by 36
  22. 2020Deep Multimodal Neural Architecture Searchcited by 34
  23. 2018Learn to Combine Modalities in Multimodal Deep Learningcited by 29
  24. 2018Light-Weight RefineNet for Real-Time Semantic Segmentationcited by 27
  25. 2016Deep Multimodal Feature Analysis for Action Recognition in RGB+D Videoscited by 26
  26. 2017Temporal Multimodal Fusion for Video Emotion Classification in the Wildcited by 18
  27. 2020Detect, Reject, Correct: Crossmodal Compensation of Corrupted Sensorscited by 17
  28. 2017Adaptive Feeding: Achieving Fast and Accurate Detections by Adaptively Combining Object Detectorscited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 8; 2024: 16; 2025: 14; 2026: 14.

54 papers.

Sort:

By area:

  1. 2024Deep Multimodal Learning with Missing Modality: A Surveycited by 89
  2. 2024Multimodal Alignment and Fusion: A Surveycited by 73
  3. 2024Multimodal Fusion on Low-quality Data: A Comprehensive Surveycited by 48
  4. 2022Behavioral Intention Prediction in Driving Scenes: A Surveycited by 30
  5. 2024Coupled Mamba: Enhanced Multi-modal Fusion with Coupled State Space Modelcited by 30
  6. 2025A Closer Look at Multimodal Representation Collapsecited by 25
  7. 2024MoME: Mixture of Multimodal Experts for Cancer Survival Predictioncited by 23
  8. 2025VLM-E2E: Enhancing End-to-End Autonomous Driving with Multimodal Driver Attention Fusioncited by 20
  9. 2024Neuro-Inspired Information-Theoretic Hierarchical Perception for Multimodal Learningcited by 18
  10. 2024Predictive Dynamic Fusioncited by 17
  11. 2024X-Fi: A Modality-Invariant Foundation Model for Multimodal Human Sensingcited by 12
  12. 2025MMQ: Multimodal Mixture-of-Quantization Tokenization for Semantic ID Generation and User Behavioral Adaptationcited by 10
38 more
  1. 2024GTP-4o: Modality-prompted Heterogeneous Graph Learning for Omni-modal Biomedical Representationcited by 7
  2. 2024Test-Time Dynamic Image Fusioncited by 6
  3. 2023Ensemble Learning for Fusion of Multiview Vision with Occlusion and Missing Information: Framework and Evaluations with Real-World Data and Applications in Driver Hand Activity Recognitioncited by 4
  4. 2023MultiZoo & MultiBench: A Standardized Toolkit for Multimodal Deep Learningcited by 4
  5. 2023Deep Equilibrium Multimodal Fusioncited by 4
  6. 2024Completed Feature Disentanglement Learning for Multimodal MRIs Analysiscited by 4
  7. 2023AdaMSS: Adaptive Multi-Modality Segmentation-to-Survival Learning for Survival Outcome Prediction from PET/CT Imagescited by 3
  8. 2025MANGO: Multimodal Attention-based Normalizing Flow Approach to Fusion Learningcited by 3
  9. 2025TableDART: Dynamic Adaptive Multi-Modal Routing for Table Understandingcited by 3
  10. 2026Inference-Time Dynamic Modality Selection for Incomplete Multimodal Classificationcited by 3
  11. 2026Multimodal Deep Generative Model for Semi-Supervised Learning under Class Imbalancecited by 3
  12. 2023Gated Driver Attention Predictorcited by 2
  13. 2024Multimodal Adaptive Inference for Document Image Classification with Anytime Early Exitingcited by 2
  14. 2024Exploring Fusion Techniques in Multimodal AI-Based Recruitment: Insights from FairCVdbcited by 2
  15. 2025OmniScene: Attention-Augmented Multimodal 4D Scene Understanding for Autonomous Drivingcited by 2
  16. 2025Multimodal Negative Learningcited by 2
  17. 2026PRAM-R: A Perception-Reasoning-Action-Memory Framework with LLM-Guided Modality Routing for Adaptive Autonomous Drivingcited by 2
  18. 2023Advancing Perception in Artificial Intelligence through Principles of Cognitive Sciencecited by 1
  19. 2024Memory-Inspired Temporal Prompt Interaction for Text-Image Classificationcited by 1
  20. 2025ADMN: A Layer-Wise Adaptive Multimodal Network for Dynamic Input Noise and Compute Resourcescited by 1
  21. 2025MVCL-DAF++: Enhancing Multimodal Intent Recognition via Prototype-Aware Contrastive Alignment and Coarse-to-Fine Dynamic Attention Fusioncited by 1
  22. 2026Addressing Missing and Noisy Modalities in One Solution: Unified Modality-Quality Framework for Low-quality Multimodal Datacited by 1
  23. 2026M3D-BFS: a Multi-stage Dynamic Fusion Strategy for Sample-Adaptive Multi-Modal Brain Network Analysiscited by 1
  24. 2022MMRNet: Improving Reliability for Multimodal Object Detection and Segmentation for Bin Picking via Multimodal Redundancycited by 0
  25. 2023Modality Influence in Multimodal Machine Learningcited by 0
  26. 2023Optimizing rgb-d semantic segmentation through multi-modal interaction and pooling attentioncited by 0
  27. 2024Progressive Alignment with VLM-LLM Feature to Augment Defect Classification for the ASE Datasetcited by 0
  28. 2024Multi-QuAD: Multi-Level Quality-Adaptive Dynamic Network for Reliable Multimodal Classificationcited by 0
  29. 2025Dig2DIG: Dig into Diffusion Information Gains for Image Fusioncited by 0
  30. 2025TACOcc:Target-Adaptive Cross-Modal Fusion with Volume Rendering for 3D Semantic Occupancycited by 0
  31. 2025EPIC: Efficient Prompt Interaction for Text-Image Classificationcited by 0
  32. 2025Learning to Route: Per-Sample Adaptive Routing for Multimodal Multitask Predictioncited by 0
  33. 2025Lyapunov-Stable Adaptive Control for Multimodal Concept Driftcited by 0
  34. 2026DCG ReID: Disentangling Collaboration and Guidance Fusion Representations for Multi-modal Vehicle Re-Identificationcited by 0
  35. 2026PID-Guided Partial Alignment for Multimodal Decentralized Federated Learningcited by 0
  36. 2026AdaFuse: Adaptive Multimodal Fusion for Lung Cancer Risk Prediction via Reinforcement Learningcited by 0
  37. 2026Early Intervention for VFM-based Multimodal Medical Image Classificationcited by 0
  38. 2026Mtextsuperscript4Fuse: Lightweight State-Space MoE with a Cross-Scale Gating Bridge for Brain Tumor Segmentationcited by 0

50 most cited of 54 All 54 (JSON)