mlsanityData to 7 Oct 2026

Geometric Multimodal Contrastive Representation Learning

arXiv 2202.03390 ↗First posted 7 Feb 2022cs.LG cs.AIcited by 40

Not on the map. Why?

Abstract (arXiv)

Learning representations of multimodal data that are both informative and robust to missing modalities at test time remains a challenging problem due to the inherent heterogeneity of data obtained from different channels. To address it, we present a novel Geometric Multimodal Contrastive (GMC) representation learning method consisting of two main components: i) a two-level architecture consisting of modality-specific base encoders, allowing to process an arbitrary number of modalities to an intermediate representation of fixed dimensionality, and a shared projection head, mapping the intermediate representations to a latent representation space; ii) a multimodal contrastive loss function that encourages the geometric alignment of the learned representations. We experimentally demonstrate that GMC representations are semantically rich and achieve state-of-the-art performance with missing modality information on three different learning problems including prediction and reinforcement learning tasks.

Key facts. Cited by 40 papers in the map (data through 7 Oct 2026). Builds on 18 of its 27 references that are in the map. First posted on arXiv 7 Feb 2022.

Builds on

18 of 27 references.

References · 18

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  2. 2013Auto-Encoding Variational Bayescited by 3,980
  3. 2015Continuous control with deep reinforcement learningcited by 3,279
  4. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,004
  5. 2019Multimodal Transformer for Unaligned Multimodal Language Sequencescited by 637
  6. 2018Multimodal Generative Models for Scalable Weakly-Supervised Learningcited by 200
  7. 2019Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Modelscited by 177
  8. 2018Learning Factorized Multimodal Representationscited by 171
  9. 2017DARLA: Improving Zero-Shot Transfer in Reinforcement Learningcited by 147
  10. 2021MultiBench: Multiscale Benchmarks for Multimodal Representation Learningcited by 143
  11. 2016Joint Multimodal Learning with Deep Generative Modelscited by 87
  12. 2021Multimodal VAE Active Inference Controllercited by 21
6 more
  1. 2021Rethinking Self-Supervised Learning: Small is Beautifulcited by 13
  2. 2019Multimodal representation models for prediction and control from partial informationcited by 11
  3. 2019Playing Games in the Dark: An approach for cross-modality transfer in reinforcement learningcited by 9
  4. 2021GeomCA: Geometric Evaluation of Data Representationscited by 9
  5. 2020Multimodal dynamics modeling for off-road autonomous vehiclescited by 6
  6. 2021How to Sense the World: Leveraging Hierarchy in Multimodal Perception for Robust Reinforcement Learning Agentscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 9; 2024: 11; 2025: 12; 2026: 6.

40 papers.

Sort:

  1. 2022A Survey of Multi-Agent Deep Reinforcement Learning with Communicationcited by 103
  2. 2023Identifiability Results for Multimodal Contrastive Learningcited by 58
  3. 2023M3AE: Multimodal Representation Learning for Brain Tumor Segmentation with Missing Modalitiescited by 56
  4. 2022Multimodal learning with graphscited by 54
  5. 2023FOCAL: Contrastive Learning for Multimodal Time-Series Sensing Signals in Factorized Orthogonal Latent Spacecited by 31
  6. 2023Learning Missing Modal Electronic Health Records with Unified Multi-modal Data Embedding and Modality-Aware Attentioncited by 25
  7. 2025A Closer Look at Multimodal Representation Collapsecited by 25
  8. 2024Cross-Modal Prototype based Multimodal Federated Learning under Severely Missing Modalitycited by 21
  9. 2023Multimodal Learning Without Labeled Multimodal Data: Guarantees and Applicationscited by 20
  10. 2023Sculpting Holistic 3D Representation in Contrastive Language-Image-3D Pre-trainingcited by 17
  11. 2024FedMAC: Tackling Partial-Modality Missing in Federated Learning with Cross-Modal Aggregation and Contrastive Regularizationcited by 16
  12. 2024Leveraging Foundation Models for Multi-modal Federated Learning with Incomplete Modalitycited by 14
28 more
  1. 2023UniS-MMC: Multimodal Classification via Unimodality-supervised Multimodal Contrastive Learningcited by 13
  2. 2023Enhanced Multimodal Representation Learning with Cross-modal KDcited by 11
  3. 2025Learning Reconfigurable Representations for Multimodal Federated Learning with Missing Datacited by 7
  4. 2025ReID5o: Achieving Omni Multi-modal Person Re-identification in a Single Modelcited by 6
  5. 2024Source-Free Cross-Modal Knowledge Transfer by Unleashing the Potential of Task-Irrelevant Datacited by 4
  6. 2024On the Efficiency and Robustness of Vibration-based Foundation Models for IoT Sensing: A Case Studycited by 4
  7. 2024Cell Morphology-Guided Small Molecule Generation with GFlowNetscited by 4
  8. 2026CyIN: Cyclic Informative Latent Space for Bridging Complete and Incomplete Multimodal Learningcited by 4
  9. 2024Overcoming Uncertain Incompleteness for Robust Multimodal Sequential Diagnosis Prediction via Curriculum Data Erasing Guided Knowledge Distillationcited by 3
  10. 2025InfoMAE: Pair-Efficient Cross-Modal Alignment for Multimodal Time-Series Sensing Signalscited by 3
  11. 2025Robult: Leveraging Redundancy and Modality Specific Features for Robust Multimodal Learningcited by 3
  12. 2026Federated Prompt-Tuning with Heterogeneous and Incomplete Multimodal Client Datacited by 3
  13. 2023DualTalker: A Cross-Modal Dual Learning Approach for Speech-Driven 3D Facial Animationcited by 2
  14. 2024Evaluation of data inconsistency for multi-modal sentiment analysiscited by 2
  15. 2024Harnessing Shared Relations via Multimodal Mixup Contrastive Learning for Multimodal Classificationcited by 2
  16. 2024Enhancing Multimodal Sentiment Analysis for Missing Modality through Self-Distillation and Unified Modality Cross-Attentioncited by 2
  17. 2025MonSTeR: a Unified Model for Motion, Scene, Text Retrievalcited by 2
  18. 2025Multi-modal Co-learning for Earth Observation: Enhancing single-modality models via modality collaborationcited by 2
  19. 2025MMiC: Mitigating Modality Incompleteness in Clustered Federated Learningcited by 1
  20. 2025The Describe-Then-Generate Bottleneck: How VLM Descriptions Alter Image Generation Outcomescited by 1
  21. 2026Addressing Missing and Noisy Modalities in One Solution: Unified Modality-Quality Framework for Low-quality Multimodal Datacited by 1
  22. 2024Foundations of Multisensory Artificial Intelligencecited by 0
  23. 2025Towards Achieving Perfect Multimodal Alignmentcited by 0
  24. 2025Are you SURE? Enhancing Multimodal Pretraining with Missing Modalities through Uncertainty Estimationcited by 0
  25. 2025Document Image Rectification Bases on Self-Adaptive Multitask Fusioncited by 0
  26. 2026Geometry of Uncertainty: Learning Metric Spaces for Multimodal State Estimation in RLcited by 0
  27. 2026Joint-Centric Dual Contrastive Alignment with Structure-Preserving and Information-Balanced Regularizationcited by 0
  28. 2026Beyond Objective Expressivity: Geometry Preservation in Multimodal Contrastive Learningcited by 0