mlsanityData to 14 Oct 2026

MMoE: Enhancing Multimodal Models with Mixtures of Multimodal Interaction Experts

arXiv 2311.09580 ↗First posted 16 Nov 2023cs.CLcited by 21

Not on the map. Why?

Abstract (arXiv)

Advances in multimodal models have greatly improved how interactions relevant to various tasks are modeled. Today's multimodal models mainly focus on the correspondence between images and text, using this for tasks like image-text matching. However, this covers only a subset of real-world interactions. Novel interactions, such as sarcasm expressed through opposing spoken words and gestures or humor expressed through utterances and tone of voice, remain challenging. In this paper, we introduce an approach to enhance multimodal models, which we call Multimodal Mixtures of Experts (MMoE). The key idea in MMoE is to train separate expert models for each type of multimodal interaction, such as redundancy present in both modalities, uniqueness in one modality, or synergy that emerges when both modalities are fused. On a sarcasm detection task (MUStARD) and a humor detection task (URFUNNY), we obtain new state-of-the-art results. MMoE is also able to be applied to various types of models to gain improvement.

Key facts. Cited by 21 papers in the map (data through 14 Oct 2026). Builds on 42 of its 75 references that are in the map. First posted on arXiv 16 Nov 2023.

Builds on

42 of 75 references.

References · 42

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  4. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  5. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  6. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  7. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  8. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  9. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
  10. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  11. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  12. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,423
30 more
  1. 2021Perceiver: General Perception with Iterative Attentioncited by 1,104
  2. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,031
  3. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  4. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
  5. 2020What Makes for Good Views for Contrastive Learning?cited by 846
  6. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 745
  7. 2019Multimodal Transformer for Unaligned Multimodal Language Sequencescited by 645
  8. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  9. 2013Learning Factored Representations in a Deep Mixture of Expertscited by 278
  10. 2018Efficient Low-rank Multimodal Fusion with Modality-Specific Factorscited by 270
  11. 2019Towards Multimodal Sarcasm Detection (An _Obviously_ Perfect Paper)cited by 174
  12. 2019UR-FUNNY: A Multimodal Language Dataset for Understanding Humorcited by 132
  13. 2023Grounding Language Models to Images for Multimodal Inputs and Outputscited by 131
  14. 2021FastMoE: A Fast Mixture-of-Expert Training Systemcited by 112
  15. 2022Do Androids Laugh at Electric Sheep? Humor "Understanding" Benchmarks from The New Yorker Caption Contestcited by 98
  16. 2023Quantifying & Modeling Multimodal Interactions: An Information Decomposition Frameworkcited by 83
  17. 2017Detecting Statistical Interactions from Neural Network Weightscited by 80
  18. 2023Factorized Contrastive Learning: Going Beyond Multi-view Redundancycited by 75
  19. 2020Does my multimodal model learn cross-modal interactions? It's harder to tell than you might think!cited by 62
  20. 2019Integrating Text and Image: Determining Multimodal Document Intent in Instagram Postscited by 37
  21. 2020Feature Interaction Interpretability: A Case for Explaining Ad-Recommendation Systems via Neural Interaction Detectioncited by 32
  22. 2022High-Modality Multimodal Transformer: Quantifying Modality & Interaction Heterogeneity for High-Modality Representation Learningcited by 31
  23. 2023MMSD2.0: Towards a Reliable Multi-modal Sarcasm Detection Systemcited by 31
  24. 2022Towards Multi-Modal Sarcasm Detection via Hierarchical Congruity Modeling with Knowledge Enhancementcited by 27
  25. 2020On Modality Bias in the TVQA Datasetcited by 24
  26. 2023Multimodal Learning Without Labeled Multimodal Data: Guarantees and Applicationscited by 21
  27. 2018Equal But Not The Same: Understanding the Implicit Relationship Between Persuasive Images and Textcited by 15
  28. 2021Feature Synergy, Redundancy, and Independence in Global Model Explanations using SHAP Vector Decompositioncited by 9
  29. 2020Voting based ensemble improves robustness of defensive modelscited by 7
  30. 2018Multi-modal Approach for Affective Computingcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 9; 2026: 11.

21 papers.

Sort:

By area:

  1. 2025I2MoE: Interpretable Multimodal Interaction-aware Mixture-of-Expertscited by 30
  2. 2025MMQ: Multimodal Mixture-of-Quantization Tokenization for Semantic ID Generation and User Behavioral Adaptationcited by 10
  3. 2025MAESTRO : Adaptive Sparse Attention and Robust Learning for Multimodal Dynamic Time Seriescited by 10
  4. 2025Massively Multimodal Foundation Models: A Framework for Capturing Interactions with Specialized Mixture-of-Expertscited by 4
  5. 2026Meta-TTRL: A Metacognitive Framework for Self-Improving Test-Time Reinforcement Learning in Unified Multimodal Modelscited by 4
  6. 2025Input Domain Aware MoE: Decoupling Routing Decisions from Task Optimization in Mixture of Expertscited by 3
  7. 2025SatireDecoder: Visual Cascaded Decoupling for Enhancing Satirical Image Comprehensioncited by 3
  8. 2026What Gets Activated: Uncovering Domain and Driver Experts in MoE Language Modelscited by 3
  9. 2025MINT: Multimodal Instruction Tuning with Multimodal Interaction Groupingcited by 2
  10. 2025Discrete Diffusion Models for Language Generationcited by 2
  11. 2026CTM-AI: A Blueprint for General AI Inspired by a Model of Consciousnesscited by 2
  12. 2026BrainFIBRE: A Foundation Model via Information Decomposition for Brain Microstructurecited by 1
9 more
  1. 2024Quantum-machine-assisted Drug Discoverycited by 0
  2. 2025Centering Emotion Hotspots: Multimodal Local-Global Fusion and Cross-Modal Alignment for Emotion Recognition in Conversationscited by 0
  3. 2026A Vision for Multisensory Intelligence: Sensing, Science, and Synergycited by 0
  4. 2026Modality-Guided Mixture of Structured Experts with Entropy-Triggered Routing for Multimodal Recommendationcited by 0
  5. 2026Mix-MoE: Improving Multilingual Machine Translation of Large Language Models through Mixed MoEscited by 0
  6. 2026Tackling Multimodal Learning Challenges with Mixture-of-Expert: A Surveycited by 0
  7. 2026Does Role Specialization Matter for Explanation Faithfulness in Mixture-of-Experts?cited by 0
  8. 2026Computational Humor with Multimodal LLMs: Methods, Datasets, Evaluation, and Challengescited by 0
  9. 2026Towards High-Level Semantic Intelligencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025MMSD3.0: A Multi-Image Benchmark for Real-World Multimodal Sarcasm Detection
  2. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities
  3. 2025Evaluating Open-Source Vision-Language Models for Multimodal Sarcasm Detection
  4. 2025Can Large Language Models Help Multimodal Language Analysis? MMLA: A Comprehensive Benchmark
  5. 2025Think Twice Before You Judge: Mixture of Dual Reasoning Experts for Multimodal Sarcasm Detection
  6. 2024CofiPara: A Coarse-to-fine Paradigm for Multimodal Sarcasm Target Identification with Large Multimodal Models
  7. 2024Mixture-of-Prompt-Experts for Multi-modal Semantic Understanding
  8. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMs