# The Costs of Pretending That There Are Data-Generating Probability Distributions in the Social World

arXiv: https://arxiv.org/abs/2407.17395 · HTML: https://mlsanity.com/paper/2407.17395/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 70 references that are in the map.
- First posted on arXiv 24 Jul 2024.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Equality of Opportunity in Supervised Learning](https://mlsanity.com/paper/1610.02413/) (2016)
- [Datasheets for Datasets](https://mlsanity.com/paper/1803.09010/) (2018)
- [Model Cards for Model Reporting](https://mlsanity.com/paper/1810.03993/) (2018)
- [Retiring Adult: New Datasets for Fair Machine Learning](https://mlsanity.com/paper/2108.04884/) (2021)
- [Algorithmic decision making and the cost of fairness](https://mlsanity.com/paper/1701.08230/) (2017)
- [Measurement and Fairness](https://mlsanity.com/paper/1912.05511/) (2019)
- [Predictive Multiplicity in Classification](https://mlsanity.com/paper/1909.06677/) (2019)
- [Towards Accountability for Machine Learning Datasets: Practices from Software Engineering and Infrastructure](https://mlsanity.com/paper/2010.13561/) (2020)
- [Predictive Multiplicity in Probabilistic Classification](https://mlsanity.com/paper/2206.01131/) (2022)
- [Difficult Lessons on Social Prediction from Wisconsin Public Schools](https://mlsanity.com/paper/2304.06205/) (2023)
- [Arbitrariness and Social Prediction: The Confounding Role of Variance in Fair Classification](https://mlsanity.com/paper/2301.11562/) (2023)
- [Swap Agnostic Learning, or Characterizing Omniprediction via Multicalibration](https://mlsanity.com/paper/2302.06726/) (2023)
- [Unprocessing Seven Years of Algorithmic Fairness](https://mlsanity.com/paper/2306.07261/) (2023)
- [Reconciling Individual Probability Forecasts](https://mlsanity.com/paper/2209.01687/) (2022)
- [The Relative Value of Prediction in Algorithmic Decision Making](https://mlsanity.com/paper/2312.08511/) (2023)
- [Allocation Requires Prediction Only if Inequality Is Low](https://mlsanity.com/paper/2406.13882/) (2024)
- [The Objective Function: Science and Society in the Age of Machine Intelligence](https://mlsanity.com/paper/2209.10418/) (2022)
- [Disentangling Model Multiplicity in Deep Learning](https://mlsanity.com/paper/2206.08890/) (2022)

## Built on by (most cited first, 4 of 4)

- [The Benchmarking Epistemology: Validity Theory for Evaluating Machine Learning Models](https://mlsanity.com/paper/2510.23191/) (2025)
- [Establishing Construct Validity in LLM Capability Benchmarks Requires Nomological Networks](https://mlsanity.com/paper/2603.15121/) (2026)
- [Reconsidering Fairness Through Unawareness From the Perspective of Model Multiplicity](https://mlsanity.com/paper/2505.16638/) (2025)
- [A Unifying Perspective on Probabilities as Model Predictions](https://mlsanity.com/paper/2609.09855/) (2026)

## Abstract (arXiv)

Machine Learning research, including work promoting fair or equitable algorithms, often relies on the concept of a data-generating probability distribution. The standard presumption is that since data points are 'sampled from' such a distribution, one can learn from observed data about this distribution and, thus, predict future data points which are also drawn from it. We argue, however, that such true probability distributions do not exist and that the rhetoric around them is harmful in social settings. We show that alternative frameworks focusing directly on relevant populations rather than abstract distributions are available and leave classical learning theory almost unchanged. Furthermore, we argue that the assumption of true probabilities or data-generating distributions can be misleading and obscure both the choices made and the goals pursued in machine learning practice. Based on these considerations, we suggest avoiding the assumption of data-generating probability distributions in the social world.