# Human-AI Collaboration for Estimating Scientific Replicability

arXiv: https://arxiv.org/abs/2605.27394 · HTML: https://mlsanity.com/paper/2605.27394/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 63 references that are in the map.
- First posted on arXiv 19 Apr 2026.

## Builds on (most cited first, 15 of 15)

- [Watch-And-Help: A Challenge for Social Perception and Human-AI Collaboration](https://mlsanity.com/paper/2010.09890/) (2020)
- [Wisdom of the Silicon Crowd: LLM Ensemble Prediction Capabilities Rival Human Crowd Accuracy](https://mlsanity.com/paper/2402.19379/) (2024)
- [Hybrid Intelligence](https://mlsanity.com/paper/2105.00691/) (2021)
- [Large Language Models Hallucination: A Comprehensive Survey](https://mlsanity.com/paper/2510.06265/) (2025)
- [The future of human-AI collaboration: a taxonomy of design knowledge for hybrid intelligence systems](https://mlsanity.com/paper/2105.03354/) (2021)
- [Limitations of Large Language Models in Clinical Problem-Solving Arising from Inflexible Reasoning](https://mlsanity.com/paper/2502.04381/) (2025)
- [Who Benchmarks the Benchmarks? Towards Comprehensive Evaluation of Commonsense Reasoning Benchmarks](https://mlsanity.com/paper/2504.07825/) (2025)
- [Soft Contamination Means Benchmarks Test Shallow Generalization](https://mlsanity.com/paper/2602.12413/) (2026)
- [Multi-period Trading Prediction Markets with Connections to Machine Learning](https://mlsanity.com/paper/1403.0648/) (2014)
- [Replication Markets: Results, Lessons, Challenges and Opportunities in AI Replication](https://mlsanity.com/paper/2005.04543/) (2020)
- [Predicting the Reproducibility of Social and Behavioral Science Papers Using Supervised Learning Models](https://mlsanity.com/paper/2104.04580/) (2021)
- [Design and Analysis of a Synthetic Prediction Market using Dynamic Convex Sets](https://mlsanity.com/paper/2101.01787/) (2021)
- [A Synthetic Prediction Market for Estimating Confidence in Published Work](https://mlsanity.com/paper/2201.06924/) (2021)
- [Designing for Reproducibility: A Qualitative Study of Challenges and Opportunities in High Energy Physics](https://mlsanity.com/paper/1903.05875/) (2019)
- [Omitted Variable Bias in Language Models Under Distribution Shift](https://mlsanity.com/paper/2602.16784/) (2026)

## Abstract (arXiv)

Determining whether published scientific findings can successfully be replicated is a long-standing challenge in the empirical sciences. Existing approaches for replicability assessment typically rely either on human judgment, i.e., creative assembly of human experts, or on machine learning models trained on paper content metadata. While both approaches have demonstrated value, each also has important limitations. Human forecasts can be influenced by cognitive biases and narrow exposure to the research literature, while automated assessments often struggle to capture contextual cues and subtle signals of credibility. In this paper, we examine a hybrid approach. Specifically, we introduce a hybrid prediction market in which algorithmic agents trade alongside human participants to jointly estimate the likelihood that a published scientific finding will be corroborated via the outcome of a controlled replication study. Agents are trained on outcomes from hundreds of prior replication studies while human participants contribute domain knowledge through real-time trading. We evaluate this hybrid approach through multiple live experiments involving participants from different academic disciplines and compare its performance to artificial-only and human-only baselines. Our results show that, except for a few cases, hybrid markets match or outperform artificial prediction markets, producing more accurate and reliable replication forecasts.