# Large Language Model-Based Agents for Automated Research Reproducibility: An Exploratory Study in Alzheimer's Disease

arXiv: https://arxiv.org/abs/2505.23852 · HTML: https://mlsanity.com/paper/2505.23852/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Medical and scientific domain agents, one of 16 directions in LLM agents and tool use.
- Builds on 16 of its 54 references that are in the map.
- First posted on arXiv 29 May 2025.

## Placed in

- LLM agents and tool use / [Medical and scientific domain agents](https://mlsanity.com/agents/medical-and-scientific-domain-agents/)

## Builds on (most cited first, 16 of 16)

- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challenges](https://mlsanity.com/paper/2401.07339/) (2024)
- [Can Large Language Models Transform Computational Social Science?](https://mlsanity.com/paper/2305.03514/) (2023)
- [Large Language Models for Generative Information Extraction: A Survey](https://mlsanity.com/paper/2312.17617/) (2023)
- [Chain of Hindsight Aligns Language Models with Feedback](https://mlsanity.com/paper/2302.02676/) (2023)
- [Empowering Biomedical Discovery with AI Agents](https://mlsanity.com/paper/2404.02831/) (2024)
- [Diagnostic Reasoning Prompts Reveal the Potential for Large Language Model Interpretability in Medicine](https://mlsanity.com/paper/2308.06834/) (2023)
- [LLMs4OL: Large Language Models for Ontology Learning](https://mlsanity.com/paper/2307.16648/) (2023)
- [Autonomous LLM-driven research from data to human-verifiable research papers](https://mlsanity.com/paper/2404.17605/) (2024)
- [Blind Judgement: Agent-Based Supreme Court Modelling With GPT](https://mlsanity.com/paper/2301.05327/) (2023)
- [The SocialAI School: Insights from Developmental Psychology Towards Artificial Socio-Cultural Agents](https://mlsanity.com/paper/2307.07871/) (2023)
- [LeafAI: query generator for clinical cohort discovery rivaling a human programmer](https://mlsanity.com/paper/2304.06203/) (2023)
- [Generalizable and Scalable Multistage Biomedical Concept Normalization Leveraging Large Language Models](https://mlsanity.com/paper/2405.15122/) (2024)
- [SIERRA: A Modular Framework for Research Automation and Reproducibility](https://mlsanity.com/paper/2208.07805/) (2022)

## Built on by (most cited first, 1 of 1)

- [IdeaAMBIG: Benchmarking Implementation-Critical Gaps in Research-Idea Specifications](https://mlsanity.com/paper/2609.10539/) (2026)

## Abstract (arXiv)

Objective: To demonstrate the capabilities of Large Language Models (LLMs) as autonomous agents to reproduce findings of published research studies using the same or similar dataset. Materials and Methods: We used the "Quick Access" dataset of the National Alzheimer's Coordinating Center (NACC). We identified highly cited published research manuscripts using NACC data and selected five studies that appeared reproducible using this dataset alone. Using GPT-4o, we created a simulated research team of LLM-based autonomous agents tasked with writing and executing code to dynamically reproduce the findings of each study, given only study Abstracts, Methods sections, and data dictionary descriptions of the dataset. Results: We extracted 35 key findings described in the Abstracts across 5 Alzheimer's studies. On average, LLM agents approximately reproduced 53.2% of findings per study. Numeric values and range-based findings often differed between studies and agents. The agents also applied statistical methods or parameters that varied from the originals, though overall trends and significance were sometimes similar. Discussion: In some cases, LLM-based agents replicated research techniques and findings. In others, they failed due to implementation flaws or missing methodological detail. These discrepancies show the current limits of LLMs in fully automating reproducibility assessments. Still, this early investigation highlights the potential of structured agent-based systems to provide scalable evaluation of scientific rigor. Conclusion: This exploratory work illustrates both the promise and limitations of LLMs as autonomous agents for automating reproducibility in biomedical research.