mlsanityData to 7 Oct 2026

Measuring the right thing: justifying metrics in AI impact assessments

arXiv 2504.05007 ↗First posted 7 Apr 2025cs.CY cs.AI cs.ETcited by 0

Not on the map. Why?

Abstract (arXiv)

AI Impact Assessments are only as good as the measures used to assess the impact of these systems. It is therefore paramount that we can justify our choice of metrics in these assessments, especially for difficult to quantify ethical and social values. We present a two-step approach to ensure metrics are properly motivated. First, a conception needs to be spelled out (e.g. Rawlsian fairness or fairness as solidarity) and then a metric can be fitted to that conception. Both steps require separate justifications, as conceptions can be judged on how well they fit with the function of, for example, fairness. We argue that conceptual engineering offers helpful tools for this step. Second, metrics need to be fitted to a conception. We illustrate this process through an examination of competing fairness metrics to illustrate that here the additional content that a conception offers helps us justify the choice for a specific metric. We thus advocate that impact assessments are not only clear on their metrics, but also on the conceptions that motivate those metrics.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 27 of its 51 references that are in the map. First posted on arXiv 7 Apr 2025.

Builds on

27 of 51 references.

References · 27

  1. 2016Variational Inference: A Review for Statisticianscited by 1,466
  2. 2013Black Box Variational Inferencecited by 317
  3. 2015Scalable Bayes via Barycenter in Wasserstein Spacecited by 44
  4. 2017Using stacking to average Bayesian predictive distributionscited by 39
  5. 2016Variational Boosting: Iteratively Refining Posterior Approximationscited by 36
  6. 2014Sampling for Inference in Probabilistic Models with Fast Bayesian Quadraturecited by 31
  7. 2013Asymptotically Exact, Embarrassingly Parallel MCMCcited by 30
  8. 2020Stacking for Non-mixing Bayesian Computations: The Curse and Blessing of Multimodal Posteriorscited by 29
  9. 2023Bayesian Synthetic Likelihoodcited by 29
  10. 2020Variational Bayesian Monte Carlo with Noisy Likelihoodscited by 23
  11. 2016Boosting Variational Inferencecited by 22
  12. 2018Variational Bayesian Monte Carlocited by 20
15 more
  1. 2022Fast Bayesian Inference with Batch Bayesian Quadrature via Kernel Recombinationcited by 20
  2. 2019Parallel Gaussian process surrogate Bayesian inference with noisy likelihood evaluationscited by 18
  3. 2016Merging MCMC Subposteriors through Gaussian-Process Approximationscited by 13
  4. 2019Universal Boosting Variational Inferencecited by 13
  5. 2013Parallelizing MCMC via Weierstrass Samplercited by 11
  6. 2022Parallel MCMC Without Embarrassing Failurescited by 11
  7. 2015Parallelizing MCMC with Random Partition Treescited by 10
  8. 2017Adaptive Gaussian process approximation for Bayesian inference with expensive likelihood functionscited by 9
  9. 2023PyVBMC: Efficient Bayesian inference in Pythoncited by 7
  10. 2022Fast and robust Bayesian Inference using Gaussian Processes with GPrycited by 5
  11. 2020Unbiased and Efficient Log-Likelihood Estimation with Inverse Binomial Samplingcited by 4
  12. 2024Bayesian Federated Learning with Hamiltonian Monte Carlo: Algorithm and Theorycited by 4
  13. 2021Approximate Bayesian inference from noisy likelihoods with Gaussian process emulated MCMCcited by 3
  14. 2021Divide-and-Conquer Fusioncited by 3
  15. 2023Fast post-process Bayesian inference with Variational Sparse Bayesian Quadraturecited by 3

Built on by

Not cited yet.