# Quantifying the uncertainty of model-based synthetic image quality metrics

arXiv: https://arxiv.org/abs/2504.03623 · HTML: https://mlsanity.com/paper/2504.03623/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 29 references that are in the map.
- First posted on arXiv 4 Apr 2025.

## Builds on (most cited first, 21 of 21)

- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Diffusion Models: A Comprehensive Survey of Methods and Applications](https://mlsanity.com/paper/2209.00796/) (2022)
- [Generative Adversarial Networks: An Overview](https://mlsanity.com/paper/1710.07035/) (2017)
- [Hands-on Bayesian Neural Networks -- a Tutorial for Deep Learning Users](https://mlsanity.com/paper/2007.06823/) (2020)
- [A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications](https://mlsanity.com/paper/2001.06937/) (2020)
- [Probabilistic Face Embeddings](https://mlsanity.com/paper/1904.09658/) (2019)
- [Pros and Cons of GAN Evaluation Measures: New Developments](https://mlsanity.com/paper/2103.09396/) (2021)
- [Improving Reconstruction Autoencoder Out-of-distribution Detection with Mahalanobis Distance](https://mlsanity.com/paper/1812.02765/) (2018)
- [Variational Learning is Effective for Large Deep Networks](https://mlsanity.com/paper/2402.17641/) (2024)
- [Data Augmentation Using GANs](https://mlsanity.com/paper/1904.09135/) (2019)
- [A Study on the Evaluation of Generative Models](https://mlsanity.com/paper/2206.10935/) (2022)
- [A Survey on GANs for Anomaly Detection](https://mlsanity.com/paper/1906.11632/) (2019)
- [Generative Adversarial Networks (GANs): An Overview of Theoretical Model, Evaluation Metrics, and Recent Developments](https://mlsanity.com/paper/2005.13178/) (2020)
- [Bayesian Metric Learning for Uncertainty Quantification in Image Retrieval](https://mlsanity.com/paper/2302.01332/) (2023)
- [A Pragmatic Note on Evaluating Generative Models with Fr'echet Inception Distance for Retinal Image Synthesis](https://mlsanity.com/paper/2502.17160/) (2025)
- [Bayesian Autoencoders for Drift Detection in Industrial Environments](https://mlsanity.com/paper/2107.13249/) (2021)
- [Generating Data using Monte Carlo Dropout](https://mlsanity.com/paper/1909.05755/) (2019)
- [Remedying uncertainty representations in visual inference through Explaining-Away Variational Autoencoders](https://mlsanity.com/paper/2404.15390/) (2024)
- [Report on the AAPM Grand Challenge on deep generative modeling for learning medical image statistics](https://mlsanity.com/paper/2405.01822/) (2024)
- [Fr'echet Denoised Distance: Enhancing Plausibility Evaluation for Generated Designs with Denoising Autoencoder](https://mlsanity.com/paper/2403.05352/) (2024)
- [Style transfer as data augmentation: evaluating unpaired image-to-image translation models in mammography](https://mlsanity.com/paper/2502.02475/) (2025)

## Built on by (most cited first, 2 of 2)

- [The FID Lottery: Quantifying Hidden Randomness in Generative-Model Evaluation](https://mlsanity.com/paper/2606.20536/) (2026)
- [Evaluating the trustworthiness of the Fr'echet Inception Distance with stochastic embedding representations](https://mlsanity.com/paper/2601.21979/) (2026)

## Abstract (arXiv)

The quality of synthetically generated images (e.g. those produced by diffusion models) are often evaluated using information about image contents encoded by pretrained auxiliary models. For example, the Fr'echet Inception Distance (FID) uses embeddings from an InceptionV3 model pretrained to classify ImageNet. The effectiveness of this feature embedding model has considerable impact on the trustworthiness of the calculated metric (affecting its suitability in several domains, including medical imaging). Here, uncertainty quantification (UQ) is used to provide a heuristic measure of the trustworthiness of the feature embedding model and an FID-like metric called the Fr'echet Autoencoder Distance (FAED). We apply Monte Carlo dropout to a feature embedding model (convolutional autoencoder) to model the uncertainty in its embeddings. The distribution of embeddings for each input are then used to compute a distribution of FAED values. We express uncertainty as the predictive variance of the embeddings as well as the standard deviation of the computed FAED values. We find that their magnitude correlates with the extent to which the inputs are out-of-distribution to the model's training data, providing some validation of its ability to assess the trustworthiness of the FAED.