# A Bias-Variance-Covariance Decomposition of Kernel Scores for Generative Models

arXiv: https://arxiv.org/abs/2310.05833 · HTML: https://mlsanity.com/paper/2310.05833/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 81 references that are in the map.
- First posted on arXiv 9 Oct 2023.

## Builds on (most cited first, 30 of 40)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks](https://mlsanity.com/paper/1511.06434/) (2015)
- [Language Models (Mostly) Know What They Know](https://mlsanity.com/paper/2207.05221/) (2022)
- [MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers](https://mlsanity.com/paper/2002.10957/) (2020)
- [Demystifying MMD GANs](https://mlsanity.com/paper/1801.01401/) (2018)
- [Text Embeddings by Weakly-Supervised Contrastive Pre-training](https://mlsanity.com/paper/2212.03533/) (2022)
- [Training Generative Adversarial Networks with Limited Data](https://mlsanity.com/paper/2006.06676/) (2020)
- [MPNet: Masked and Permuted Pre-training for Language Understanding](https://mlsanity.com/paper/2004.09297/) (2020)
- [Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation](https://mlsanity.com/paper/2302.09664/) (2023)
- [Towards General Text Embeddings with Multi-stage Contrastive Learning](https://mlsanity.com/paper/2308.03281/) (2023)
- [CoQA: A Conversational Question Answering Challenge](https://mlsanity.com/paper/1808.07042/) (2018)
- [Perception Prioritized Training of Diffusion Models](https://mlsanity.com/paper/2204.00227/) (2022)
- [Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Search](https://mlsanity.com/paper/2005.11129/) (2020)
- [Generative Moment Matching Networks](https://mlsanity.com/paper/1502.02761/) (2015)
- [MMD GAN: Towards Deeper Understanding of Moment Matching Network](https://mlsanity.com/paper/1705.08584/) (2017)
- [Unsupervised Quality Estimation for Neural Machine Translation](https://mlsanity.com/paper/2005.10608/) (2020)
- [Training generative neural networks via Maximum Mean Discrepancy optimization](https://mlsanity.com/paper/1505.03906/) (2015)
- [Diverse Weight Averaging for Out-of-Distribution Generalization](https://mlsanity.com/paper/2205.09739/) (2022)
- [A Kernel Test of Goodness of Fit](https://mlsanity.com/paper/1602.02964/) (2016)
- [Learning Deep Kernels for Non-Parametric Two-Sample Tests](https://mlsanity.com/paper/2002.09116/) (2020)
- [Assessing Generalization of SGD via Disagreement](https://mlsanity.com/paper/2106.13799/) (2021)
- [Consistent View Synthesis with Pose-Guided Diffusion Models](https://mlsanity.com/paper/2303.17598/) (2023)
- [Kernels for sequentially ordered data](https://mlsanity.com/paper/1601.08169/) (2016)

## Built on by (most cited first, 8 of 8)

- [Fine-Grained Uncertainty Decomposition in Large Language Models: A Spectral Approach](https://mlsanity.com/paper/2509.22272/) (2025)
- [Inv-Entropy: A Fully Probabilistic Framework for Uncertainty Quantification in Language Models](https://mlsanity.com/paper/2506.09684/) (2025)
- [Disentangling Ambiguity from Instability in Large Language Models: A Clinical Text-to-SQL Case Study](https://mlsanity.com/paper/2602.12015/) (2026)
- [Variability Need Not Imply Error: The Case of Adequate but Semantically Distinct Responses](https://mlsanity.com/paper/2412.15683/) (2024)
- [Uncertainty Quantification for Regression: A Unified Framework based on kernel scores](https://mlsanity.com/paper/2510.25599/) (2025)
- [SceneNAT: Masked Generative Modeling for Language-Guided Indoor Scene Synthesis](https://mlsanity.com/paper/2601.07218/) (2026)
- [Disentangling Mean Embeddings for Better Diagnostics of Image Generators](https://mlsanity.com/paper/2409.01314/) (2024)
- [Eigenvalue Calibration for Semantic Embeddings of Large Language Models](https://mlsanity.com/paper/2607.08377/) (2026)

## Abstract (arXiv)

Generative models, like large language models, are becoming increasingly relevant in our daily lives, yet a theoretical framework to assess their generalization behavior and uncertainty does not exist. Particularly, the problem of uncertainty estimation is commonly solved in an ad-hoc and task-dependent manner. For example, natural language approaches cannot be transferred to image generation. In this paper, we introduce the first bias-variance-covariance decomposition for kernel scores. This decomposition represents a theoretical framework from which we derive a kernel-based variance and entropy for uncertainty estimation. We propose unbiased and consistent estimators for each quantity which only require generated samples but not the underlying model itself. Based on the wide applicability of kernels, we demonstrate our framework via generalization and uncertainty experiments for image, audio, and language generation. Specifically, kernel entropy for uncertainty estimation is more predictive of performance on CoQA and TriviaQA question answering datasets than existing baselines and can also be applied to closed-source models.