mlsanityData to 14 Oct 2026

MM-Align: Learning Optimal Transport-based Alignment Dynamics for Fast and Accurate Inference on Missing Modality Sequences

arXiv 2210.12798 ↗First posted 23 Oct 2022cs.CL cs.AI cs.LGcited by 8

Not on the map. Why?

Abstract (arXiv)

Existing multimodal tasks mostly target at the complete input modality setting, i.e., each modality is either complete or completely missing in both training and test sets. However, the randomly missing situations have still been underexplored. In this paper, we present a novel approach named MM-Align to address the missing-modality inference problem. Concretely, we propose 1) an alignment dynamics learning module based on the theory of optimal transport (OT) for indirect missing data imputation; 2) a denoising training algorithm to simultaneously enhance the imputation results and backbone network performance. Compared with previous methods which devote to reconstructing the missing inputs, MM-Align learns to capture and imitate the alignment dynamics between modality sequences. Results of comprehensive experiments on three datasets covering two multimodal tasks empirically demonstrate that our method can perform more accurate and faster inference and relieve overfitting under various missing conditions.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 32 of its 54 references that are in the map. First posted on arXiv 23 Oct 2022.

Builds on

32 of 54 references.

References · 32

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  4. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
  5. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  6. 2017Prototypical Networks for Few-shot Learningcited by 3,015
  7. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,367
  8. 2017Learning to Compare: Relation Network for Few-Shot Learningcited by 1,143
  9. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,013
  10. 2019Multimodal Transformer for Unaligned Multimodal Language Sequencescited by 645
  11. 2018MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversationscited by 643
  12. 2020Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need?cited by 437
20 more
  1. 2017Tensor Fusion Network for Multimodal Sentiment Analysiscited by 433
  2. 2018Meta-Transfer Learning for Few-Shot Learningcited by 306
  3. 2017Toward Multimodal Image-to-Image Translationcited by 296
  4. 2018Efficient Low-rank Multimodal Fusion with Modality-Specific Factorscited by 270
  5. 2021SMIL: Multimodal Learning with Severely Missing Modalitycited by 245
  6. 2021Few-shot Image Generation via Cross-domain Correspondencecited by 207
  7. 2018Learning Factorized Multimodal Representationscited by 171
  8. 2018Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalitiescited by 149
  9. 2021Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysiscited by 131
  10. 2016Dual Attention Networks for Multimodal Reasoning and Matchingcited by 114
  11. 2020Few-Shot Text Generation with Pattern-Exploiting Trainingcited by 87
  12. 2014Curriculum Learning of Multiple Taskscited by 66
  13. 2020Few-Shot Adaptation of Generative Adversarial Networkscited by 64
  14. 2019Dense Classification and Implanting for Few-Shot Learningcited by 56
  15. 2021MIRACLE: Causally-Aware Imputation via Learning Missing Data Mechanismscited by 51
  16. 2021Bi-Bimodal Modality Fusion for Correlation-Controlled Multimodal Sentiment Analysiscited by 45
  17. 2021Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignmentcited by 41
  18. 2020TransModality: An End2End Fusion Method with Transformer for Multimodal Sentiment Analysiscited by 40
  19. 2019Learning Representations from Imperfect Time Series Data via Tensor Rank Regularizationcited by 30
  20. 2021Improving Multimodal fusion via Mutual Dependency Maximisationcited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 2; 2025: 2; 2026: 2.

8 papers.

  1. 2023Robust Multimodal Learning with Missing Modalities via Parameter-Efficient Adaptationcited by 29
  2. 2024Toward Robust Multimodal Learning using Multimodal Foundational Modelscited by 4
  3. 2024Towards Robust Instruction Tuning on Multimodal Large Language Modelscited by 2
  4. 2025Towards Robust and Realible Multimodal Misinformation Recognition with Incomplete Modalitycited by 2
  5. 2023Modality-Collaborative Transformer with Hybrid Feature Reconstruction for Robust Emotion Recognitioncited by 1
  6. 2025Robust Multimodal Sentiment Analysis of Image-Text Pairs by Distribution-Based Feature Recovery and Fusioncited by 1
  7. 2026MissMAC-Bench: Building Solid Benchmark for Missing Modality Issue in Robust Multimodal Affective Computingcited by 1
  8. 2026Bridging the Missing-Modality Gap: Improving Text-Only Calibration of Vision Language Modelscited by 0