# What Level of Automation is "Good Enough"? A Benchmark of Large Language Models for Meta-Analysis Data Extraction

arXiv: https://arxiv.org/abs/2507.15152 · HTML: https://mlsanity.com/paper/2507.15152/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 61 references that are in the map.
- First posted on arXiv 20 Jul 2025.

## Builds on (most cited first, 11 of 11)

- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [Agentic Reasoning: A Streamlined Framework for Enhancing LLM Reasoning with Agentic Tools](https://mlsanity.com/paper/2502.04644/) (2025)
- [Match, Compare, or Select? An Investigation of Large Language Models for Entity Matching](https://mlsanity.com/paper/2405.16884/) (2024)
- [How Interpretable are Reasoning Explanations from Prompting Large Language Models?](https://mlsanity.com/paper/2402.11863/) (2024)
- [When Hindsight is Not 20/20: Testing Limits on Reflective Thinking in Large Language Models](https://mlsanity.com/paper/2404.09129/) (2024)
- [TART: An Open-Source Tool-Augmented Framework for Explainable Table-based Reasoning](https://mlsanity.com/paper/2409.11724/) (2024)
- [Automatically Extracting Numerical Results from Randomized Controlled Trials with Large Language Models](https://mlsanity.com/paper/2405.01686/) (2024)
- [Exploring the use of a Large Language Model for data extraction in systematic reviews: a rapid feasibility study](https://mlsanity.com/paper/2405.14445/) (2024)
- [Transforming Evidence Synthesis: A Systematic Review of the Evolution of Automated Meta-Analysis in the Age of AI](https://mlsanity.com/paper/2504.20113/) (2025)
- [Evaluating Search Engines and Large Language Models for Answering Health Questions](https://mlsanity.com/paper/2407.12468/) (2024)
- [AI-Assisted Data Extraction for Systematic Reviews in Education](https://mlsanity.com/paper/2501.11840/) (2025)

## Built on by (most cited first, 6 of 6)

- [Compiling Prompts, Not Crafting Them: A Reproducible Workflow for AI-Assisted Evidence Synthesis](https://mlsanity.com/paper/2509.00038/) (2025)
- [AutoSynthesis: An agentic system for automated meta-analysis](https://mlsanity.com/paper/2607.15247/) (2026)
- [A Reproducible Optimisation Protocol for Calibrating Prompt-Based Large Language Model Workflows in Evidence Synthesis](https://mlsanity.com/paper/2605.06937/) (2026)
- [Constructing Executable Analytical Knowledge Representations for Meta-Analysis Synthesis Using an Agentic Harness](https://mlsanity.com/paper/2608.01711/) (2026)
- [Using Human-LLM Disagreement to Improve Checklist-Based Quality Appraisal](https://mlsanity.com/paper/2608.20385/) (2026)
- [EviStreams: Human-in-the-Loop AI Data Extraction for Systematic Reviews in Medicine](https://mlsanity.com/paper/2609.27418/) (2026)

## Abstract (arXiv)

Automating data extraction from full-text randomised controlled trials (RCTs) for meta-analysis remains a significant challenge. This study evaluates the practical performance of three LLMs (Gemini-2.0-flash, Grok-3, GPT-4o-mini) across tasks involving statistical results, risk-of-bias assessments, and study-level characteristics in three medical domains: hypertension, diabetes, and orthopaedics. We tested four distinct prompting strategies (basic prompting, self-reflective prompting, model ensemble, and customised prompts) to determine how to improve extraction quality. All models demonstrate high precision but consistently suffer from poor recall by omitting key information. We found that customised prompts were the most effective, boosting recall by up to 15%. Based on this analysis, we propose a three-tiered set of guidelines for using LLMs in data extraction, matching data types to appropriate levels of automation based on task complexity and risk. Our study offers practical advice for automating data extraction in real-world meta-analyses, balancing LLM efficiency with expert oversight through targeted, task-specific automation.