# Modality to Modality Translation: An Adversarial Representation Learning and Graph Fusion Network for Multimodal Fusion

arXiv: https://arxiv.org/abs/1911.07848 · HTML: https://mlsanity.com/paper/1911.07848/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Nov 2019.

## Built on by (most cited first, 30 of 45)

- [Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2109.00412/) (2021)
- [GCNet: Graph Completion Network for Incomplete Multimodal Learning in Conversation](https://mlsanity.com/paper/2203.02177/) (2022)
- [Multimodal Alignment and Fusion: A Survey](https://mlsanity.com/paper/2411.17040/) (2024)
- [UniMSE: Towards Unified Multimodal Sentiment Analysis and Emotion Recognition](https://mlsanity.com/paper/2211.11256/) (2022)
- [Multimodal learning with graphs](https://mlsanity.com/paper/2209.03299/) (2022)
- [Multimodal Information Bottleneck: Learning Minimal Sufficient Unimodal and Multimodal Representations](https://mlsanity.com/paper/2210.17444/) (2022)
- [Hybrid Contrastive Learning of Tri-Modal Representation for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2109.01797/) (2021)
- [Multimodal Fusion and Vision-Language Models: A Survey for Robot Vision](https://mlsanity.com/paper/2504.02477/) (2025)
- [Learnable Expansion of Graph Operators for Multi-Modal Feature Fusion](https://mlsanity.com/paper/2410.01506/) (2024)
- [Tailor Versatile Multi-modal Learning for Multi-label Emotion Recognition](https://mlsanity.com/paper/2201.05834/) (2022)
- [Counterfactual Reasoning for Out-of-distribution Multimodal Sentiment Analysis](https://mlsanity.com/paper/2207.11652/) (2022)
- [UniSA: Unified Generative Framework for Sentiment Analysis](https://mlsanity.com/paper/2309.01339/) (2023)
- [Large Language Models Meet Text-Centric Multimodal Sentiment Analysis: A Survey](https://mlsanity.com/paper/2406.08068/) (2024)
- [Revisiting Disentanglement and Fusion on Modality and Context in Conversational Multimodal Emotion Recognition](https://mlsanity.com/paper/2308.04502/) (2023)
- [Recent Advances in Multimodal Affective Computing: An NLP Perspective](https://mlsanity.com/paper/2409.07388/) (2024)
- [ScaleVLAD: Improving Multimodal Sentiment Analysis via Multi-Scale Fusion of Locally Descriptors](https://mlsanity.com/paper/2112.01368/) (2021)
- [SANCL: Multimodal Review Helpfulness Prediction with Selective Attention and Natural Contrastive Learning](https://mlsanity.com/paper/2209.05040/) (2022)
- [General Debiasing for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2307.10511/) (2023)
- [Multimodal Fusion Refiner Networks](https://mlsanity.com/paper/2104.03435/) (2021)
- [Trustworthy Multimodal Fusion for Sentiment Analysis in Ordinal Sentiment Space](https://mlsanity.com/paper/2404.08923/) (2024)
- [Sentiment Word Aware Multimodal Refinement for Multimodal Sentiment Analysis with ASR Errors](https://mlsanity.com/paper/2203.00257/) (2022)
- [AudioVisual Video Summarization](https://mlsanity.com/paper/2105.07667/) (2021)
- [Which is Making the Contribution: Modulating Unimodal and Cross-modal Dynamics for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2111.08451/) (2021)
- [Deep Biological Pathway Informed Pathology-Genomic Multimodal Survival Prediction](https://mlsanity.com/paper/2301.02383/) (2023)
- [Ring That Bell: A Corpus and Method for Multimodal Metaphor Detection in Videos](https://mlsanity.com/paper/2301.01134/) (2022)
- [Multimodal Graph Representation Learning for Robust Surgical Workflow Recognition with Adversarial Feature Disentanglement](https://mlsanity.com/paper/2505.01766/) (2025)
- [RoHyDR: Robust Hybrid Diffusion Recovery for Incomplete Multimodal Emotion Recognition](https://mlsanity.com/paper/2505.17501/) (2025)
- [Video Sentiment Analysis with Bimodal Information-augmented Multi-Head Attention](https://mlsanity.com/paper/2103.02362/) (2021)
- [Disentangling Bias by Modeling Intra- and Inter-modal Causal Attention for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2508.04999/) (2025)
- [Hardness-Aware Dynamic Curriculum Learning for Robust Multimodal Emotion Recognition with Missing Modalities](https://mlsanity.com/paper/2508.06800/) (2025)

## Abstract (arXiv)

Learning joint embedding space for various modalities is of vital importance for multimodal fusion. Mainstream modality fusion approaches fail to achieve this goal, leaving a modality gap which heavily affects cross-modal fusion. In this paper, we propose a novel adversarial encoder-decoder-classifier framework to learn a modality-invariant embedding space. Since the distributions of various modalities vary in nature, to reduce the modality gap, we translate the distributions of source modalities into that of target modality via their respective encoders using adversarial training. Furthermore, we exert additional constraints on embedding space by introducing reconstruction loss and classification loss. Then we fuse the encoded representations using hierarchical graph neural network which explicitly explores unimodal, bimodal and trimodal interactions in multi-stage. Our method achieves state-of-the-art performance on multiple datasets. Visualization of the learned embeddings suggests that the joint embedding space learned by our method is discriminative. code is available at: urlhttps://github.com/TmacMai/ARGFmultimodalfusion