# Automated reproducibility assessments in the social and behavioral sciences using large language models

arXiv: https://arxiv.org/abs/2606.13670 · HTML: https://mlsanity.com/paper/2606.13670/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 24 references that are in the map.
- First posted on arXiv 11 Jun 2026.

## Builds on (most cited first, 5 of 5)

- [ChatDev: Communicative Agents for Software Development](https://mlsanity.com/paper/2307.07924/) (2023)
- [Towards End-to-End Automation of AI Research](https://mlsanity.com/paper/2606.15497/) (2026)
- [The Good, The Bad, and The Greedy: Evaluation of LLMs Should Not Ignore Non-Determinism](https://mlsanity.com/paper/2407.10457/) (2024)
- [LAMBDA: A Large Model Based Data Agent](https://mlsanity.com/paper/2407.17535/) (2024)
- [StatLLM: A Dataset for Evaluating the Performance of Large Language Models in Statistical Analysis](https://mlsanity.com/paper/2502.17657/) (2025)

## Built on by (most cited first, 1 of 1)

- [AutoSynthesis: An agentic system for automated meta-analysis](https://mlsanity.com/paper/2607.15247/) (2026)

## Abstract (arXiv)

Reproducibility in the social and behavioral sciences is typically evaluated by independent researchers who reanalyze the original data to assess whether the published findings can be recovered. However, such approaches are resource-intensive and difficult to scale. Here, we show that large language models (LLMs) can automate reproducibility assessments. Using N = 180 published studies with predefined claims from the behavioral and social sciences, we compare LLM-generated analyses with the original findings. For 11 studies, the LLM pipeline could not produce a viable effect size estimate. For the remaining studies, the LLM reached the same qualitative conclusion as the original study in 80% of cases, and recovered the original effect sizes (using a +/-0.05 tolerance in Cohen's d) in 24% of studies. In a subset with human reanalyses, the LLM reached the same qualitative conclusion as the original study in 95% of studies, similar to human reanalysts (83%), and the LLM recovered the original effect sizes using a +/-0.05 tolerance in 40% of studies, again broadly similar to human reanalysts (28%). Given the current capabilities and limitations of LLMs, the findings show that LLMs can support systematic audits of empirical results rather than substitute expert judgment. As such, LLMs can serve as a scalable screening tool to improve the rigor and reproducibility in empirical research.