# Measuring the right thing: justifying metrics in AI impact assessments

arXiv: https://arxiv.org/abs/2504.05007 · HTML: https://mlsanity.com/paper/2504.05007/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 27 of its 51 references that are in the map.
- First posted on arXiv 7 Apr 2025.

## Builds on (most cited first, 27 of 27)

- [Variational Inference: A Review for Statisticians](https://mlsanity.com/paper/1601.00670/) (2016)
- [Black Box Variational Inference](https://mlsanity.com/paper/1401.0118/) (2013)
- [Scalable Bayes via Barycenter in Wasserstein Space](https://mlsanity.com/paper/1508.05880/) (2015)
- [Using stacking to average Bayesian predictive distributions](https://mlsanity.com/paper/1704.02030/) (2017)
- [Variational Boosting: Iteratively Refining Posterior Approximations](https://mlsanity.com/paper/1611.06585/) (2016)
- [Sampling for Inference in Probabilistic Models with Fast Bayesian Quadrature](https://mlsanity.com/paper/1411.0439/) (2014)
- [Asymptotically Exact, Embarrassingly Parallel MCMC](https://mlsanity.com/paper/1311.4780/) (2013)
- [Stacking for Non-mixing Bayesian Computations: The Curse and Blessing of Multimodal Posteriors](https://mlsanity.com/paper/2006.12335/) (2020)
- [Bayesian Synthetic Likelihood](https://mlsanity.com/paper/2305.05120/) (2023)
- [Variational Bayesian Monte Carlo with Noisy Likelihoods](https://mlsanity.com/paper/2006.08655/) (2020)
- [Boosting Variational Inference](https://mlsanity.com/paper/1611.05559/) (2016)
- [Variational Bayesian Monte Carlo](https://mlsanity.com/paper/1810.05558/) (2018)
- [Fast Bayesian Inference with Batch Bayesian Quadrature via Kernel Recombination](https://mlsanity.com/paper/2206.04734/) (2022)
- [Parallel Gaussian process surrogate Bayesian inference with noisy likelihood evaluations](https://mlsanity.com/paper/1905.01252/) (2019)
- [Merging MCMC Subposteriors through Gaussian-Process Approximations](https://mlsanity.com/paper/1605.08576/) (2016)
- [Universal Boosting Variational Inference](https://mlsanity.com/paper/1906.01235/) (2019)
- [Parallelizing MCMC via Weierstrass Sampler](https://mlsanity.com/paper/1312.4605/) (2013)
- [Parallel MCMC Without Embarrassing Failures](https://mlsanity.com/paper/2202.11154/) (2022)
- [Parallelizing MCMC with Random Partition Trees](https://mlsanity.com/paper/1506.03164/) (2015)
- [Adaptive Gaussian process approximation for Bayesian inference with expensive likelihood functions](https://mlsanity.com/paper/1703.09930/) (2017)
- [PyVBMC: Efficient Bayesian inference in Python](https://mlsanity.com/paper/2303.09519/) (2023)
- [Fast and robust Bayesian Inference using Gaussian Processes with GPry](https://mlsanity.com/paper/2211.02045/) (2022)
- [Unbiased and Efficient Log-Likelihood Estimation with Inverse Binomial Sampling](https://mlsanity.com/paper/2001.03985/) (2020)
- [Bayesian Federated Learning with Hamiltonian Monte Carlo: Algorithm and Theory](https://mlsanity.com/paper/2407.06935/) (2024)
- [Approximate Bayesian inference from noisy likelihoods with Gaussian process emulated MCMC](https://mlsanity.com/paper/2104.03942/) (2021)
- [Divide-and-Conquer Fusion](https://mlsanity.com/paper/2110.07265/) (2021)
- [Fast post-process Bayesian inference with Variational Sparse Bayesian Quadrature](https://mlsanity.com/paper/2303.05263/) (2023)

## Abstract (arXiv)

AI Impact Assessments are only as good as the measures used to assess the impact of these systems. It is therefore paramount that we can justify our choice of metrics in these assessments, especially for difficult to quantify ethical and social values. We present a two-step approach to ensure metrics are properly motivated. First, a conception needs to be spelled out (e.g. Rawlsian fairness or fairness as solidarity) and then a metric can be fitted to that conception. Both steps require separate justifications, as conceptions can be judged on how well they fit with the function of, for example, fairness. We argue that conceptual engineering offers helpful tools for this step. Second, metrics need to be fitted to a conception. We illustrate this process through an examination of competing fairness metrics to illustrate that here the additional content that a conception offers helps us justify the choice for a specific metric. We thus advocate that impact assessments are not only clear on their metrics, but also on the conceptions that motivate those metrics.