mlsanityData to 7 Oct 2026

Multimodal Transformer for Unaligned Multimodal Language Sequences

arXiv 1906.00295 ↗First posted 1 Jun 2019cs.CLcited by 637

Not on the map. Why?

Abstract (arXiv)

Human language is often multimodal, which comprehends a mixture of natural language, facial gestures, and acoustic behaviors. However, two major challenges in modeling such multimodal human language time-series data exist: 1) inherent data non-alignment due to variable sampling rates for the sequences from each modality; and 2) long-range dependencies between elements across modalities. In this paper, we introduce the Multimodal Transformer (MulT) to generically address the above issues in an end-to-end manner without explicitly aligning the data. At the heart of our model is the directional pairwise crossmodal attention, which attends to interactions between multimodal sequences across distinct time steps and latently adapt streams from one modality to another. Comprehensive experiments on both aligned and non-aligned multimodal time-series show that our model outperforms state-of-the-art methods by a large margin. In addition, empirical analysis suggests that correlated crossmodal signals are able to be captured by the proposed crossmodal attention mechanism in MulT.

Key facts. Cited by 637 papers in the map (data through 7 Oct 2026). First posted on arXiv 1 Jun 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 58; 2022: 96; 2023: 101; 2024: 108; 2025: 143; 2026: 131.

637 papers.

Sort:

By area:

  1. 2022Multimodal Learning with Transformers: A Surveycited by 399
  2. 2022Are Multimodal Transformers Robust to Missing Modality?cited by 183
  3. 2021Learning Modality-Specific Representations with Self-Supervised Multi-Task Learning for Multimodal Sentiment Analysiscited by 179
  4. 2022Trusted Multi-View Classification with Dynamic Evidential Fusioncited by 167
  5. 2021MultiBench: Multiscale Benchmarks for Multimodal Representation Learningcited by 143
  6. 2022TransTab: Learning Transferable Tabular Transformers Across Tablescited by 132
  7. 2021Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysiscited by 130
  8. 2021CDTrans: Cross-domain Transformer for Unsupervised Domain Adaptationcited by 129
  9. 2022Learning to Listen: Modeling Non-Deterministic Dyadic Facial Motioncited by 125
  10. 2022Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questionscited by 116
  11. 2021Rethinking Architecture Design for Tackling Data Heterogeneity in Federated Learningcited by 111
  12. 2021MMGCN: Multimodal Fusion via Deep Graph Convolution Network for Emotion Recognition in Conversationcited by 99
38 more
  1. 2023Modeling Dense Multimodal Interactions Between Biological Pathways and Histology for Survival Predictioncited by 92
  2. 2023Retrieving Multimodal Information for Augmented Generation: A Surveycited by 84
  3. 2022Mind's Eye: Grounded Language Model Reasoning through Simulationcited by 83
  4. 2023Quantifying & Modeling Multimodal Interactions: An Information Decomposition Frameworkcited by 81
  5. 2023Decoupled Multimodal Distilling for Emotion Recognitioncited by 76
  6. 2021Learning Vision-Guided Quadrupedal Locomotion End-to-End with Cross-Modal Transformerscited by 71
  7. 2022M3Care: Learning with Missing Modalities in Multimodal Healthcare Datacited by 71
  8. 2024FuseMoE: Mixture-of-Experts Transformers for Fleximodal Fusioncited by 67
  9. 2022UniMSE: Towards Unified Multimodal Sentiment Analysis and Emotion Recognitioncited by 61
  10. 2023Identifiability Results for Multimodal Contrastive Learningcited by 58
  11. 2022Multimodal Information Bottleneck: Learning Minimal Sufficient Unimodal and Multimodal Representationscited by 48
  12. 2023SimMMDG: A Simple and Effective Framework for Multi-modal Domain Generalizationcited by 48
  13. 2022M2FNet: Multi-modal Fusion Network for Emotion Recognition in Conversationcited by 47
  14. 2021Bi-Bimodal Modality Fusion for Correlation-Controlled Multimodal Sentiment Analysiscited by 45
  15. 2022Visual Acoustic Matchingcited by 45
  16. 2024Towards a Foundation Model for Partial Differential Equations: Multi-Operator Learning and Extrapolationcited by 45
  17. 2021MM-ViT: Multi-Modal Video Transformer for Compressed Video Action Recognitioncited by 44
  18. 2022Make Acoustic and Visual Cues Matter: CH-SIMS v2.0 Dataset and AV-Mixup Consistent Modulecited by 43
  19. 2022Visuo-Tactile Transformers for Manipulationcited by 43
  20. 2025AffectGPT: A New Dataset, Model, and Benchmark for Emotion Understanding with Multimodal Large Language Modelscited by 43
  21. 2021Visual Grounding with Transformerscited by 42
  22. 2021Hybrid Contrastive Learning of Tri-Modal Representation for Multimodal Sentiment Analysiscited by 41
  23. 2022MIntRec: A New Dataset for Multimodal Intent Recognitioncited by 41
  24. 2021Multimodal End-to-End Sparse Model for Emotion Recognitioncited by 40
  25. 2021Feature Fusion Vision Transformer for Fine-Grained Visual Categorizationcited by 40
  26. 2022Geometric Multimodal Contrastive Representation Learningcited by 40
  27. 2022CubeMLP: An MLP-based Model for Multimodal Sentiment Analysis and Depression Estimationcited by 39
  28. 2023Transformers in Speech Processing: A Surveycited by 38
  29. 2023GPT-4V with Emotion: A Zero-shot Benchmark for Generalized Emotion Recognitioncited by 38
  30. 2023Multimodal Variational Auto-encoder based Audio-Visual Segmentationcited by 37
  31. 2023A Transformer-Based Model With Self-Distillation for Multimodal Emotion Recognition in Conversationscited by 37
  32. 2025Multimodal Fusion and Vision-Language Models: A Survey for Robot Visioncited by 37
  33. 2021Multimodal Learning using Optimal Transport for Sarcasm and Humor Detectioncited by 36
  34. 2022MultiViz: Towards Visualizing and Understanding Multimodal Modelscited by 36
  35. 2021Multi-modal Text Recognition Networks: Interactive Enhancements between Visual and Semantic Featurescited by 34
  36. 2024Classifier-guided Gradient Modulation for Enhanced Multimodal Learningcited by 34
  37. 2022Improving Medical Predictions by Irregular Multimodal Electronic Health Records Modelingcited by 33
  38. 2021Vision Guided Generative Pre-trained Language Models for Multimodal Abstractive Summarizationcited by 32

50 most cited of 637 All 637 (JSON)