# General Bayesian Loss Function Selection and the use of Improper Models

arXiv: https://arxiv.org/abs/2106.01214 · HTML: https://mlsanity.com/paper/2106.01214/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Jun 2021.

## Built on by (most cited first, 14 of 14)

- [Outlier-robust Kalman Filtering through Generalised Bayes](https://mlsanity.com/paper/2405.05646/) (2024)
- [Robust and Scalable Bayesian Online Changepoint Detection](https://mlsanity.com/paper/2302.04759/) (2023)
- [Generalised Bayesian Inference for Discrete Intractable Likelihood](https://mlsanity.com/paper/2206.08420/) (2022)
- [Robust and Conjugate Gaussian Process Regression](https://mlsanity.com/paper/2311.00463/) (2023)
- [Concentration properties of fractional posterior in 1-bit matrix completion](https://mlsanity.com/paper/2404.08969/) (2024)
- [A sparse PAC-Bayesian approach for high-dimensional quantile prediction](https://mlsanity.com/paper/2409.01687/) (2024)
- [Multi-Output Robust and Conjugate Gaussian Processes](https://mlsanity.com/paper/2510.26401/) (2025)
- [Minimizing robust density power-based divergences for general parametric density models](https://mlsanity.com/paper/2307.05251/) (2023)
- [Adaptive posterior concentration rates for sparse high-dimensional linear regression with random design and unknown error variance](https://mlsanity.com/paper/2405.19016/) (2024)
- [Correntropy-Based Improper Likelihood Model for Robust Electrophysiological Source Imaging](https://mlsanity.com/paper/2408.14843/) (2024)
- [Locking and Quacking: Stacking Bayesian model predictions by log-pooling and superposition](https://mlsanity.com/paper/2305.07334/) (2023)
- [Robust Sparse Bayesian Learning Based on Minimum Error Entropy for Noisy High-Dimensional Brain Activity Decoding](https://mlsanity.com/paper/2508.11657/) (2025)
- [Amortized Bandwidth Learning for Kernel Density Estimation under Logarithmic Score](https://mlsanity.com/paper/2608.20445/) (2026)
- [Robust Bayesian Inference for Unnormalized Models with Mixed-Domain Data](https://mlsanity.com/paper/2609.01783/) (2026)

## Abstract (arXiv)

Statisticians often face the choice between using probability models or a paradigm defined by minimising a loss function. Both approaches are useful and, if the loss can be re-cast into a proper probability model, there are many tools to decide which model or loss is more appropriate for the observed data, in the sense of explaining the data's nature. However, when the loss leads to an improper model, there are no principled ways to guide this choice. We address this task by combining the Hyv"arinen score, which naturally targets infinitesimal relative probabilities, and general Bayesian updating, which provides a unifying framework for inference on losses and models. Specifically we propose the H-score, a general Bayesian selection criterion and prove that it consistently selects the (possibly improper) model closest to the data-generating truth in Fisher's divergence. We also prove that an associated H-posterior consistently learns optimal hyper-parameters featuring in loss functions, including a challenging tempering parameter in generalised Bayesian inference. As salient examples, we consider robust regression and non-parametric density estimation where popular loss functions define improper models for the data and hence cannot be dealt with using standard model selection tools. These examples illustrate advantages in robustness-efficiency trade-offs and provide a Bayesian implementation for kernel density estimation, opening a new avenue for Bayesian non-parametrics.