# CM-GANs: Cross-modal Generative Adversarial Networks for Common Representation Learning

arXiv: https://arxiv.org/abs/1710.05106 · HTML: https://mlsanity.com/paper/1710.05106/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Oct 2017.

## Built on by (most cited first, 27 of 27)

- [Multimodal Co-learning: Challenges, Applications with Datasets, Recent Advances and Future Directions](https://mlsanity.com/paper/2107.13782/) (2021)
- [Fold2Seq: A Joint Sequence(1D)-Fold(3D) Embedding-based Generative Model for Protein Design](https://mlsanity.com/paper/2106.13058/) (2021)
- [Integrating Language Guidance into Vision-based Deep Metric Learning](https://mlsanity.com/paper/2203.08543/) (2022)
- [Multi-modality Associative Bridging through Memory: Speech Sound Recollected from Face Video](https://mlsanity.com/paper/2204.01265/) (2022)
- [Progressive Localization Networks for Language-based Moment Localization](https://mlsanity.com/paper/2102.01282/) (2021)
- [AKVSR: Audio Knowledge Empowered Visual Speech Recognition by Compressing Audio Knowledge of a Pretrained Model](https://mlsanity.com/paper/2308.07593/) (2023)
- [ScaleVLAD: Improving Multimodal Sentiment Analysis via Multi-Scale Fusion of Locally Descriptors](https://mlsanity.com/paper/2112.01368/) (2021)
- [Exploring The Effect of High-frequency Components in GANs Training](https://mlsanity.com/paper/2103.11093/) (2021)
- [MVAR: Visual Autoregressive Modeling with Scale and Spatial Markovian Conditioning](https://mlsanity.com/paper/2505.12742/) (2025)
- [GAME-ON: Graph Attention Network based Multimodal Fusion for Fake News Detection](https://mlsanity.com/paper/2202.12478/) (2022)
- [A SAM-guided Two-stream Lightweight Model for Anomaly Detection](https://mlsanity.com/paper/2402.19145/) (2024)
- [PHyCLIP: ℓ1-Product of Hyperbolic Factors Unifies Hierarchy and Compositionality in Vision-Language Representation Learning](https://mlsanity.com/paper/2510.08919/) (2025)
- [Auxiliary Cross-Modal Representation Learning with Triplet Loss Functions for Online Handwriting Recognition](https://mlsanity.com/paper/2202.07901/) (2022)
- [CDKT-FL: Cross-Device Knowledge Transfer using Proxy Dataset in Federated Learning](https://mlsanity.com/paper/2204.01542/) (2022)
- [LILE: Look In-Depth before Looking Elsewhere -- A Dual Attention Network using Transformers for Cross-Modal Information Retrieval in Histopathology Archives](https://mlsanity.com/paper/2203.01445/) (2022)
- [A Survey of Multimodal Composite Editing and Retrieval](https://mlsanity.com/paper/2409.05405/) (2024)
- [A Comprehensive Empirical Study of Vision-Language Pre-trained Model for Supervised Cross-Modal Retrieval](https://mlsanity.com/paper/2201.02772/) (2022)
- [Long-term, Short-term and Sudden Event: Trading Volume Movement Prediction with Graph-based Multi-view Modeling](https://mlsanity.com/paper/2108.11318/) (2021)
- [Trace Ratio Optimization with an Application to Multi-view Learning](https://mlsanity.com/paper/2101.04292/) (2021)
- [An Unsupervised Sampling Approach for Image-Sentence Matching Using Document-Level Structural Information](https://mlsanity.com/paper/2104.02605/) (2021)
- [A Survey of Recent Advances and Challenges in Deep Audio-Visual Correlation Learning](https://mlsanity.com/paper/2412.00049/) (2024)
- [Self-Augmented Multi-Modal Feature Embedding](https://mlsanity.com/paper/2103.04731/) (2021)
- [Recent Advances and Challenges in Deep Audio-Visual Correlation Learning](https://mlsanity.com/paper/2202.13673/) (2022)
- [Research on Cross-media Science and Technology Information Data Retrieval](https://mlsanity.com/paper/2204.04887/) (2022)
- [CRC-RL: A Novel Visual Feature Representation Architecture for Unsupervised Reinforcement Learning](https://mlsanity.com/paper/2301.13473/) (2023)
- [LEARN: A Story-Driven Layout-to-Image Generation Framework for STEM Instruction](https://mlsanity.com/paper/2508.11153/) (2025)
- [Cross-Modal Retrieval with Cauchy-Schwarz Divergence](https://mlsanity.com/paper/2509.21339/) (2025)

## Abstract (arXiv)

It is known that the inconsistent distribution and representation of different modalities, such as image and text, cause the heterogeneity gap that makes it challenging to correlate such heterogeneous data. Generative adversarial networks (GANs) have shown its strong ability of modeling data distribution and learning discriminative representation, existing GANs-based works mainly focus on generative problem to generate new data. We have different goal, aim to correlate heterogeneous data, by utilizing the power of GANs to model cross-modal joint distribution. Thus, we propose Cross-modal GANs to learn discriminative common representation for bridging heterogeneity gap. The main contributions are: (1) Cross-modal GANs architecture is proposed to model joint distribution over data of different modalities. The inter-modality and intra-modality correlation can be explored simultaneously in generative and discriminative models. Both of them beat each other to promote cross-modal correlation learning. (2) Cross-modal convolutional autoencoders with weight-sharing constraint are proposed to form generative model. They can not only exploit cross-modal correlation for learning common representation, but also preserve reconstruction information for capturing semantic consistency within each modality. (3) Cross-modal adversarial mechanism is proposed, which utilizes two kinds of discriminative models to simultaneously conduct intra-modality and inter-modality discrimination. They can mutually boost to make common representation more discriminative by adversarial training process. To the best of our knowledge, our proposed CM-GANs approach is the first to utilize GANs to perform cross-modal common representation learning. Experiments are conducted to verify the performance of our proposed approach on cross-modal retrieval paradigm, compared with 10 methods on 3 cross-modal datasets.