# Proper losses for discrete generative models

arXiv: https://arxiv.org/abs/2211.03761 · HTML: https://mlsanity.com/paper/2211.03761/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 24 references that are in the map.
- First posted on arXiv 7 Nov 2022.

## Builds on (most cited first, 8 of 8)

- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Demystifying MMD GANs](https://mlsanity.com/paper/1801.01401/) (2018)
- [A note on the evaluation of generative models](https://mlsanity.com/paper/1511.01844/) (2015)
- [Generative Moment Matching Networks](https://mlsanity.com/paper/1502.02761/) (2015)
- [Pros and Cons of GAN Evaluation Measures](https://mlsanity.com/paper/1802.03446/) (2018)
- [Training generative neural networks via Maximum Mean Discrepancy optimization](https://mlsanity.com/paper/1505.03906/) (2015)
- [Toward a Characterization of Loss Functions for Distribution Learning](https://mlsanity.com/paper/1906.02652/) (2019)
- [Multi-Observation Elicitation](https://mlsanity.com/paper/1706.01394/) (2017)

## Built on by (most cited first, 4 of 4)

- [ElicitationGPT: Text Elicitation Mechanisms via Language Models](https://mlsanity.com/paper/2406.09363/) (2024)
- [A Bias-Variance Decomposition for Ensembles over Multiple Synthetic Datasets](https://mlsanity.com/paper/2402.03985/) (2024)
- [Aligned Textual Scoring Rules](https://mlsanity.com/paper/2507.06221/) (2025)
- [Distribution Learning with Valid Outputs Beyond the Worst-Case](https://mlsanity.com/paper/2410.16253/) (2024)

## Abstract (arXiv)

We initiate the study of proper losses for evaluating generative models in the discrete setting. Unlike traditional proper losses, we treat both the generative model and the target distribution as black-boxes, only assuming ability to draw i.i.d. samples. We define a loss to be black-box proper if the generative distribution that minimizes expected loss is equal to the target distribution. Using techniques from statistical estimation theory, we give a general construction and characterization of black-box proper losses: they must take a polynomial form, and the number of draws from the model and target distribution must exceed the degree of the polynomial. The characterization rules out a loss whose expectation is the cross-entropy between the target distribution and the model. By extending the construction to arbitrary sampling schemes such as Poisson sampling, however, we show that one can construct such a loss.