mlsanityData to 7 Oct 2026

Gated Multimodal Units for Information Fusion

arXiv 1702.01992 ↗First posted 7 Feb 2017stat.ML cs.LGcited by 244

Not on the map. Why?

Abstract (arXiv)

This paper presents a novel model for multimodal learning based on gated neural networks. The Gated Multimodal Unit (GMU) model is intended to be used as an internal unit in a neural network architecture whose purpose is to find an intermediate representation based on a combination of data from different modalities. The GMU learns to decide how modalities influence the activation of the unit using multiplicative gates. It was evaluated on a multilabel scenario for genre classification of movies using the plot and the poster. The GMU improved the macro f-score performance of single-modality approaches and outperformed other fusion strategies, including mixture of experts models. Along with this work, the MM-IMDb dataset is released which, to the best of our knowledge, is the largest publicly available multimodal dataset for genre prediction on movies.

Key facts. Cited by 244 papers in the map (data through 7 Oct 2026). First posted on arXiv 7 Feb 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 23; 2022: 25; 2023: 27; 2024: 41; 2025: 47; 2026: 81.

244 papers.

Sort:

By area:

  1. 2021SMIL: Multimodal Learning with Severely Missing Modalitycited by 243
  2. 2022Versatile Diffusion: Text, Images and Variations All in One Diffusion Modelcited by 199
  3. 2022Are Multimodal Transformers Robust to Missing Modality?cited by 183
  4. 2022A Survey of Vision-Language Pre-Trained Modelscited by 177
  5. 2021MultiBench: Multiscale Benchmarks for Multimodal Representation Learningcited by 143
  6. 2023Multimodal Prompting with Missing Modalities for Visual Recognitioncited by 121
  7. 2022Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questionscited by 116
  8. 2022Hate-CLIPper: Multimodal Hateful Meme Classification based on Cross-modal Interaction of CLIP Featurescited by 55
  9. 2022Dynamic Multimodal Fusioncited by 54
  10. 2022Shifting More Attention to Visual Backbone: Query-modulated Refinement Networks for End-to-End Visual Groundingcited by 52
  11. 2021A Survey on Multimodal Disinformation Detectioncited by 47
  12. 2022The Modality Focusing Hypothesis: Towards Understanding Crossmodal Knowledge Distillationcited by 44
38 more
  1. 2023Surgical-VQLA: Transformer with Gated Vision-Language Embedding for Visual Question Localized-Answering in Robotic Surgerycited by 44
  2. 2021A Survey on Graph-Based Deep Learning for Computational Histopathologycited by 41
  3. 2024What to align in multimodal contrastive learning?cited by 41
  4. 2021AdaMML: Adaptive Multi-Modal Learning for Efficient Video Recognitioncited by 36
  5. 2022MultiViz: Towards Visualizing and Understanding Multimodal Modelscited by 36
  6. 2021Multimodal Co-learning: Challenges, Applications with Datasets, Recent Advances and Future Directionscited by 35
  7. 2021Multi-modal Sarcasm Detection and Humor Classification in Code-mixed Conversationscited by 33
  8. 2023Efficient Multimodal Fusion via Interactive Promptingcited by 32
  9. 2024Coupled Mamba: Enhanced Multi-modal Fusion with Coupled State Space Modelcited by 30
  10. 2021Fusion and Orthogonal Projection for Improved Face-Voice Associationcited by 29
  11. 2025I2MoE: Interpretable Multimodal Interaction-aware Mixture-of-Expertscited by 29
  12. 2025Understanding the Emergence of Multimodal Representation Alignmentcited by 28
  13. 2023Multi-modal Machine Learning in Engineering Design: A Review and Future Directionscited by 27
  14. 2022Self-supervised Implicit Glyph Attention for Text Recognitioncited by 25
  15. 2024HEMM: Holistic Evaluation of Multimodal Foundation Modelscited by 21
  16. 2022Defending Multimodal Fusion Models against Single-Source Adversariescited by 20
  17. 2024Missing Modality Prediction for Unpaired Multimodal Learning via Joint Embedding of Unimodal Modelscited by 20
  18. 2021Revisiting spatio-temporal layouts for compositional action recognitioncited by 19
  19. 2024Play to the Score: Stage-Guided Dynamic Multi-Sensory Fusion for Robotic Manipulationcited by 19
  20. 2023CAT-ViL: Co-Attention Gated Vision-Language Embedding for Visual Question Localized-Answering in Robotic Surgerycited by 17
  21. 2024Surgical-VQLA++: Adversarial Contrastive Learning for Calibrated Robust Visual Question-Localized Answering in Robotic Surgerycited by 17
  22. 2024Deep Correlated Prompting for Visual Recognition with Missing Modalitiescited by 17
  23. 2022Group Gated Fusion on Attention-based Bidirectional Alignment for Multimodal Emotion Recognitioncited by 16
  24. 2023Towards Robust Multimodal Prompting With Missing Modalitiescited by 16
  25. 2022Learnable Irrelevant Modality Dropout for Multimodal Action Recognition on Modality-Specific Annotated Videoscited by 14
  26. 2021BM-NAS: Bilevel Multimodal Neural Architecture Searchcited by 13
  27. 2024Dovetail: A CPU/GPU Heterogeneous Speculative Decoding for LLM inferencecited by 13
  28. 2025Explainable Action Form Assessment by Exploiting Multimodal Chain-of-Thoughts Reasoningcited by 13
  29. 2025FlowerDance: MeanFlow for Efficient and Refined 3D Dance Generationcited by 10
  30. 2021Detection of Illicit Drug Trafficking Events on Instagram: A Deep Multimodal Multilabel Learning Approachcited by 9
  31. 2025Knowledge Bridger: Towards Training-free Missing Modality Completioncited by 9
  32. 2025See-Saw Modality Balance: See Gradient, and Sew Impaired Vision-Language Balance to Mitigate Dominant Modality Biascited by 9
  33. 2025PAEFF: Precise Alignment and Enhanced Gated Feature Fusion for Face-Voice Associationcited by 9
  34. 2021Multimodal Fusion Refiner Networkscited by 8
  35. 2022Multimodal Tree Decoder for Table of Contents Extraction in Document Imagescited by 8
  36. 2023Boosting Crop Classification by Hierarchically Fusing Satellite, Rotational, and Contextual Datacited by 8
  37. 2023Harmonic-NAS: Hardware-Aware Multimodal Neural Architecture Search on Resource-constrained Devicescited by 8
  38. 2025Synergistic Prompting for Robust Visual Recognition with Missing Modalitiescited by 8

50 most cited of 244 All 244 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Multi-Layer Visual Feature Fusion in Multimodal LLMs: Methods, Analysis, and Best Practices
  2. 2025Demystifying ChatGPT: How It Masters Genre Recognition
  3. 2025Predicting Brain Responses To Natural Movies With Multimodal LLMs
  4. 2025Towards LLM-Centric Multimodal Fusion: A Survey on Integration Strategies and Techniques
  5. 2025MM-MovieDubber: Towards Multi-Modal Learning for Multi-Modal Movie Dubbing
  6. 2024Multimodal Large Language Models and Tunings: Vision, Language, Sensors, Audio, and Beyond
  7. 2024On the generalization capacity of neural networks during generic multimodal reasoning
  8. 2025ModRWKV: Transformer Multimodality in Linear Time