# A Test of Relative Similarity For Model Selection in Generative Models

arXiv: https://arxiv.org/abs/1511.04581 · HTML: https://mlsanity.com/paper/1511.04581/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Nov 2015.

## Built on by (most cited first, 25 of 25)

- [Deep Learning Approaches for Data Augmentation in Medical Imaging: A Review](https://mlsanity.com/paper/2307.13125/) (2023)
- [Evaluation Metrics for Graph Generative Models: Problems, Pitfalls, and Practical Solutions](https://mlsanity.com/paper/2106.01098/) (2021)
- [Can You Rely on Your Model Evaluation? Improving Model Evaluation with Synthetic Test Data](https://mlsanity.com/paper/2310.16524/) (2023)
- [Implicit Generative Copulas](https://mlsanity.com/paper/2109.14567/) (2021)
- [Composite Goodness-of-fit Tests with Kernels](https://mlsanity.com/paper/2111.10275/) (2021)
- [A survey on GANs for computer vision: Recent research, analysis and taxonomy](https://mlsanity.com/paper/2203.11242/) (2022)
- [Ten Years of Generative Adversarial Nets (GANs): A survey of the state-of-the-art](https://mlsanity.com/paper/2308.16316/) (2023)
- [Generating Physically-Consistent Satellite Imagery for Climate Visualizations](https://mlsanity.com/paper/2104.04785/) (2021)
- [MMD-B-Fair: Learning Fair Representations with Statistical Testing](https://mlsanity.com/paper/2211.07907/) (2022)
- [Reducing Training Sample Memorization in GANs by Training with Memorization Rejection](https://mlsanity.com/paper/2210.12231/) (2022)
- [Variance Matters: Improving Domain Adaptation via Stratified Sampling](https://mlsanity.com/paper/2512.05226/) (2025)
- [Evaluating Disentanglement in Generative Models Without Knowledge of Latent Factors](https://mlsanity.com/paper/2210.01760/) (2022)
- [Kernel-Based Tests for Likelihood-Free Hypothesis Testing](https://mlsanity.com/paper/2308.09043/) (2023)
- [Risk Aware Benchmarking of Large Language Models](https://mlsanity.com/paper/2310.07132/) (2023)
- [Defining 'Good': Evaluation Framework for Synthetic Smart Meter Data](https://mlsanity.com/paper/2407.11785/) (2024)
- [A synthetic dataset of French electric load curves with temperature conditioning](https://mlsanity.com/paper/2504.14046/) (2025)
- [Regularized f-Divergence Kernel Tests](https://mlsanity.com/paper/2601.19755/) (2026)
- [Magnitude Distance: A Geometric Measure of Dataset Similarity](https://mlsanity.com/paper/2602.08859/) (2026)
- [Kernel Tests of Equivalence](https://mlsanity.com/paper/2603.10886/) (2026)
- [Statistical Inference for Generative Model Comparison](https://mlsanity.com/paper/2501.18897/) (2025)
- [Keep your distance: learning dispersed embeddings on mathbbSm](https://mlsanity.com/paper/2502.08231/) (2025)
- [ReLATE: Resilient Learner Selection for Multivariate Time-Series Classification Against Adversarial Attacks](https://mlsanity.com/paper/2503.07882/) (2025)
- [Anchor-based Maximum Discrepancy for Relative Similarity Testing](https://mlsanity.com/paper/2510.10477/) (2025)
- [Learning Robust Representations Of Generative Models Using Set-Based Artificial Fingerprints](https://mlsanity.com/paper/2206.02067/) (2022)
- [From Vessel Trajectories to Safety-Critical Encounter Scenarios: A Generative AI Framework for Autonomous Ship Digital Testing](https://mlsanity.com/paper/2603.28067/) (2026)

## Abstract (arXiv)

Probabilistic generative models provide a powerful framework for representing data that avoids the expense of manual annotation typically needed by discriminative approaches. Model selection in this generative setting can be challenging, however, particularly when likelihoods are not easily accessible. To address this issue, we introduce a statistical test of relative similarity, which is used to determine which of two models generates samples that are significantly closer to a real-world reference dataset of interest. We use as our test statistic the difference in maximum mean discrepancies (MMDs) between the reference dataset and each model dataset, and derive a powerful, low-variance test based on the joint asymptotic distribution of the MMDs between each reference-model pair. In experiments on deep generative models, including the variational auto-encoder and generative moment matching network, the tests provide a meaningful ranking of model performance as a function of parameter and training settings.