# TCGM: An Information-Theoretic Framework for Semi-Supervised Multi-Modality Learning

arXiv: https://arxiv.org/abs/2007.06793 · HTML: https://mlsanity.com/paper/2007.06793/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Jul 2020.

## Built on by (most cited first, 19 of 19)

- [What Makes Multi-modal Learning Better than Single (Provably)](https://mlsanity.com/paper/2106.04538/) (2021)
- [Modality Competition: What Makes Joint Training of Multi-modal Network Fail in Deep Learning? (Provably)](https://mlsanity.com/paper/2203.12221/) (2022)
- [The Modality Focusing Hypothesis: Towards Understanding Crossmodal Knowledge Distillation](https://mlsanity.com/paper/2206.06487/) (2022)
- [Understanding Multimodal Contrastive Learning and Incorporating Unpaired Data](https://mlsanity.com/paper/2302.06232/) (2023)
- [On the Generalization of Multi-modal Contrastive Learning](https://mlsanity.com/paper/2306.04272/) (2023)
- [Multimodal Knowledge Expansion](https://mlsanity.com/paper/2103.14431/) (2021)
- [A Theory of Multimodal Learning](https://mlsanity.com/paper/2309.12458/) (2023)
- [Learning Multimodal Data Augmentation in Feature Space](https://mlsanity.com/paper/2212.14453/) (2022)
- [STiL: Semi-supervised Tabular-Image Learning for Comprehensive Task-Relevant Information Exploration in Multimodal Classification](https://mlsanity.com/paper/2503.06277/) (2025)
- [SoundingActions: Learning How Actions Sound from Narrated Egocentric Videos](https://mlsanity.com/paper/2404.05206/) (2024)
- [What Makes for Robust Multi-Modal Models in the Face of Missing Modalities?](https://mlsanity.com/paper/2310.06383/) (2023)
- [Variational Distillation for Multi-View Learning](https://mlsanity.com/paper/2206.09548/) (2022)
- [Greedy Modality Selection via Approximate Submodular Maximization](https://mlsanity.com/paper/2210.12562/) (2022)
- [Training-Free Robust Multimodal Learning via Sample-Wise Jacobian Regularization](https://mlsanity.com/paper/2204.02485/) (2022)
- [Towards Multimodal Domain Generalization with Few Labels](https://mlsanity.com/paper/2602.22917/) (2026)
- [Incomplete Multimodal Learning for Complex Brain Disorders Prediction](https://mlsanity.com/paper/2305.16222/) (2023)
- [On the Computational Benefit of Multimodal Learning](https://mlsanity.com/paper/2309.13782/) (2023)
- [Uncertainty-Aware Multimodal Learning via Conformal Shapley Intervals](https://mlsanity.com/paper/2602.00171/) (2026)
- [On Stronger Computational Separations Between Multimodal and Unimodal Machine Learning](https://mlsanity.com/paper/2404.02254/) (2024)

## Abstract (arXiv)

Fusing data from multiple modalities provides more information to train machine learning systems. However, it is prohibitively expensive and time-consuming to label each modality with a large amount of data, which leads to a crucial problem of semi-supervised multi-modal learning. Existing methods suffer from either ineffective fusion across modalities or lack of theoretical guarantees under proper assumptions. In this paper, we propose a novel information-theoretic approach, namely Total Correlation Gain Maximization (TCGM), for semi-supervised multi-modal learning, which is endowed with promising properties: (i) it can utilize effectively the information across different modalities of unlabeled data points to facilitate training classifiers of each modality (ii) it has theoretical guarantee to identify Bayesian classifiers, i.e., the ground truth posteriors of all modalities. Specifically, by maximizing TC-induced loss (namely TC gain) over classifiers of all modalities, these classifiers can cooperatively discover the equivalent class of ground-truth classifiers; and identify the unique ones by leveraging limited percentage of labeled data. We apply our method to various tasks and achieve state-of-the-art results, including news classification, emotion recognition and disease prediction.