# Information Theoretic Co-Training

arXiv: https://arxiv.org/abs/1802.07572 · HTML: https://mlsanity.com/paper/1802.07572/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Feb 2018.

## Built on by (most cited first, 7 of 7)

- [Decomposed Mutual Information Estimation for Contrastive Representation Learning](https://mlsanity.com/paper/2106.13401/) (2021)
- [Autoregressive Co-Training for Learning Discrete Speech Representations](https://mlsanity.com/paper/2203.15840/) (2022)
- [Revisiting Self-supervised Learning of Speech Representation from a Mutual Information Perspective](https://mlsanity.com/paper/2401.08833/) (2024)
- [MICO: Selective Search with Mutual Information Co-training](https://mlsanity.com/paper/2209.04378/) (2022)
- [Inducing Systematicity in Transformers by Attending to Structurally Quantized Embeddings](https://mlsanity.com/paper/2402.06492/) (2024)
- [Review of Unsupervised POS Tagging and Its Implications on Language Acquisition](https://mlsanity.com/paper/2312.10169/) (2023)
- [Learning Speech Representations with Variational Predictive Coding](https://mlsanity.com/paper/2601.00100/) (2025)

## Abstract (arXiv)

This paper introduces an information theoretic co-training objective for unsupervised learning. We consider the problem of predicting the future. Rather than predict future sensations (image pixels or sound waves) we predict "hypotheses" to be confirmed by future sensations. More formally, we assume a population distribution on pairs (x,y) where we can think of x as a past sensation and y as a future sensation. We train both a predictor model P_Phi(z|x) and a confirmation model P_Psi(z|y) where we view z as hypotheses (when predicted) or facts (when confirmed). For a population distribution on pairs (x,y) we focus on the problem of measuring the mutual information between x and y. By the data processing inequality this mutual information is at least as large as the mutual information between x and z under the distribution on triples (x,z,y) defined by the confirmation model P_Psi(z|y). The information theoretic training objective for P_Phi(z|x) and P_Psi(z|y) can be viewed as a form of co-training where we want the prediction from x to match the confirmation from y.