# On distinguishability criteria for estimating generative models

arXiv: https://arxiv.org/abs/1412.6515 · HTML: https://mlsanity.com/paper/1412.6515/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Dec 2014.

## Built on by (most cited first, 21 of 21)

- [Decision-Focused Learning: Foundations, State of the Art, Benchmark and Future Opportunities](https://mlsanity.com/paper/2307.13565/) (2023)
- [Machine Learning for Synthetic Data Generation: A Review](https://mlsanity.com/paper/2302.04062/) (2023)
- [Tailoring Language Generation Models under Total Variation Distance](https://mlsanity.com/paper/2302.13344/) (2023)
- [Generative Adversarial Neural Cellular Automata](https://mlsanity.com/paper/2108.04328/) (2021)
- [Time-series Generation by Contrastive Imitation](https://mlsanity.com/paper/2311.01388/) (2023)
- [Fast Variational AutoEncoder with Inverted Multi-Index for Collaborative Filtering](https://mlsanity.com/paper/2109.05773/) (2021)
- [A Quantum Generative Adversarial Network for distributions](https://mlsanity.com/paper/2110.02742/) (2021)
- [DiffImpute: Tabular Data Imputation With Denoising Diffusion Probabilistic Model](https://mlsanity.com/paper/2403.13863/) (2024)
- [Statistical applications of contrastive learning](https://mlsanity.com/paper/2204.13999/) (2022)
- [Generative Adversarial Networks and Adversarial Autoencoders: Tutorial and Survey](https://mlsanity.com/paper/2111.13282/) (2021)
- [Generative Adversarial Networks for Image Augmentation in Agriculture: A Systematic Review](https://mlsanity.com/paper/2204.04707/) (2022)
- [Harnessing Synthetic Data from Generative AI for Statistical Inference](https://mlsanity.com/paper/2603.05396/) (2026)
- [A Decade of Deep Learning for Remote Sensing Spatiotemporal Fusion: Advances, Challenges, and Opportunities](https://mlsanity.com/paper/2504.00901/) (2025)
- [Evaluating Synthetic Tabular Data Generated To Augment Small Sample Datasets](https://mlsanity.com/paper/2211.10760/) (2022)
- [A Physics-Informed Low-Shot Learning For sEMG-Based Estimation of Muscle Force and Joint Kinematics](https://mlsanity.com/paper/2307.05361/) (2023)
- [On the connection between Noise-Contrastive Estimation and Contrastive Divergence](https://mlsanity.com/paper/2402.16688/) (2024)
- [Adversarial Training for a Hybrid Approach to Aspect-Based Sentiment Analysis](https://mlsanity.com/paper/2111.14988/) (2021)
- [Shared Loss between Generators of GANs](https://mlsanity.com/paper/2211.07234/) (2022)
- [Task-Agnostic Graph Neural Network Evaluation via Adversarial Collaboration](https://mlsanity.com/paper/2301.11517/) (2023)
- [Controlling Face's Frame generation in StyleGAN's latent space operations: Modifying faces to deceive our memory](https://mlsanity.com/paper/2407.00803/) (2024)
- [Integrating AIRL-Based Reward Estimation into a Control Method Using Fuzzy Inference and Policy Gradient](https://mlsanity.com/paper/2610.14723/) (2026)

## Abstract (arXiv)

Two recently introduced criteria for estimation of generative models are both based on a reduction to binary classification. Noise-contrastive estimation (NCE) is an estimation procedure in which a generative model is trained to be able to distinguish data samples from noise samples. Generative adversarial networks (GANs) are pairs of generator and discriminator networks, with the generator network learning to generate samples by attempting to fool the discriminator network into believing its samples are real data. Both estimation procedures use the same function to drive learning, which naturally raises questions about how they are related to each other, as well as whether this function is related to maximum likelihood estimation (MLE). NCE corresponds to training an internal data model belonging to the em discriminator network but using a fixed generator network. We show that a variant of NCE, with a dynamic generator network, is equivalent to maximum likelihood estimation. Since pairing a learned discriminator with an appropriate dynamically selected generator recovers MLE, one might expect the reverse to hold for pairing a learned generator with a certain discriminator. However, we show that recovering MLE for a learned generator requires departing from the distinguishability game. Specifically: (i) The expected gradient of the NCE discriminator can be made to match the expected gradient of MLE, if one is allowed to use a non-stationary noise distribution for NCE, (ii) No choice of discriminator network can make the expected gradient for the GAN generator match that of MLE, and (iii) The existing theory does not guarantee that GANs will converge in the non-convex case. This suggests that the key next step in GAN research is to determine whether GANs converge, and if not, to modify their training algorithm to force convergence.