# Defining Replicability of Prediction Rules

arXiv: https://arxiv.org/abs/2305.01518 · HTML: https://mlsanity.com/paper/2305.01518/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 58 references that are in the map.
- First posted on arXiv 30 Apr 2023.

## Builds on (most cited first, 10 of 10)

- [WILDS: A Benchmark of in-the-Wild Distribution Shifts](https://mlsanity.com/paper/2012.07421/) (2020)
- [A Comprehensive Survey on Transfer Learning](https://mlsanity.com/paper/1911.02685/) (2019)
- [Generalizing to Unseen Domains: A Survey on Domain Generalization](https://mlsanity.com/paper/2103.03097/) (2021)
- [An introduction to domain adaptation and transfer learning](https://mlsanity.com/paper/1812.11806/) (2018)
- [Veridical Data Science](https://mlsanity.com/paper/1901.08152/) (2019)
- [Terminologies for Reproducible Research](https://mlsanity.com/paper/1802.03311/) (2018)
- [Improving the repeatability of deep learning models with Monte Carlo dropout](https://mlsanity.com/paper/2202.07562/) (2022)
- [Quantifying replicability and consistency in systematic reviews](https://mlsanity.com/paper/1907.06856/) (2019)
- [Integration of Survival Data from Multiple Studies](https://mlsanity.com/paper/2007.08594/) (2020)
- [Deciding whether follow-up studies have replicated findings in a preliminary large-scale "omics' study"](https://mlsanity.com/paper/1310.0606/) (2013)

## Abstract (arXiv)

In this article I propose an approach for defining replicability for prediction rules. Motivated by a recent NAS report, I start from the perspective that replicability is obtaining consistent results across studies suitable to address the same prediction question, each of which has obtained its own data. I then discuss concept and issues in defining key elements of this statement. I focus specifically on the meaning of "consistent results" in typical utilization contexts, and propose a multi-agent framework for defining replicability, in which agents are neither partners nor adversaries. I recover some of the prevalent practical approaches as special cases. I hope to provide guidance for a more systematic assessment of replicability in machine learning.