mlsanityData to 7 Oct 2026

DRDF: Determining the Importance of Different Multimodal Information with Dual-Router Dynamic Framework

arXiv 2107.09909 ↗First posted 21 Jul 2021cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

In multimodal tasks, we find that the importance of text and image modal information is different for different input cases, and for this motivation, we propose a high-performance and highly general Dual-Router Dynamic Framework (DRDF), consisting of Dual-Router, MWF-Layer, experts and expert fusion unit. The text router and image router in Dual-Router accept text modal information and image modal information, and use MWF-Layer to determine the importance of modal information. Based on the result of the determination, MWF-Layer generates fused weights for the fusion of experts. Experts are model backbones that match the current task. DRDF has high performance and high generality, and we have tested 12 backbones such as Visual BERT on multimodal dataset Hateful memes, unimodal dataset CIFAR10, CIFAR100, and TinyImagenet. Our DRDF outperforms all the baselines. We also verified the components of DRDF in detail by ablations, compared and discussed the reasons and ideas of DRDF design.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 24 of its 28 references that are in the map. First posted on arXiv 21 Jul 2021.

Builds on

24 of 28 references.

References · 24

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  4. 2017Squeeze-and-Excitation Networkscited by 4,763
  5. 2016Wide Residual Networkscited by 3,057
  6. 2018CBAM: Convolutional Block Attention Modulecited by 2,463
  7. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  8. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  9. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  10. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  11. 2020The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memescited by 485
  12. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
12 more
  1. 2021Dynamic Neural Networks: A Surveycited by 405
  2. 2019Selective Kernel Networkscited by 401
  3. 2019Dynamic Convolution: Attention over Convolution Kernelscited by 355
  4. 2019CondConv: Conditionally Parameterized Convolutions for Efficient Inferencecited by 273
  5. 2017Gated Multimodal Units for Information Fusioncited by 244
  6. 2016SCA-CNN: Spatial and Channel-wise Attention in Convolutional Networks for Image Captioningcited by 219
  7. 2019Supervised Multimodal Bitransformers for Classifying Images and Textcited by 130
  8. 2017Attention-Based Multimodal Fusion for Video Descriptioncited by 75
  9. 2019Squeeze-and-Attention Networks for Semantic Segmentationcited by 60
  10. 2020NBDT: Neural-Backed Decision Treescited by 46
  11. 2019Deformable Kernels: Adapting Effective Receptive Fields for Object Deformationcited by 29
  12. 2020Long-Tailed Recognition Using Class-Balanced Expertscited by 10

Built on by

Not cited yet.