# AI-Assisted Data Extraction for Systematic Reviews in Education

arXiv: https://arxiv.org/abs/2501.11840 · HTML: https://mlsanity.com/paper/2501.11840/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Jan 2025.

## Built on by (most cited first, 3 of 3)

- [What Level of Automation is "Good Enough"? A Benchmark of Large Language Models for Meta-Analysis Data Extraction](https://mlsanity.com/paper/2507.15152/) (2025)
- [From Automation to Collaboration: Human-in-the-Loop Methods for Safe and Trustworthy NLP](https://mlsanity.com/paper/2605.25226/) (2026)
- [Large Language Models as Supervised Extraction Assistants: Lowering the Barrier to Documentation Standard Adoption in Agent-Based Modelling](https://mlsanity.com/paper/2606.13749/) (2026)

## Abstract (arXiv)

Systematic reviews are time-consuming endeavors that require knowledgeable human reviewers to screen studies for relevance and extract data following a specific coding scheme before any analysis or synthesis can occur. Large language models (LLMs) hold promise for substantially accelerating this process and reducing reviewer workload, yet their application within the context of systematic reviews in the field of education remains underexplored. We address this issue in two ways: through empirical studies and the iterative development of an open-source software tool. First, we conducted two empirical studies examining the efficacy of using LLMs for data extraction using data from a published review on pedagogical agents. We extracted a variety of data types from 112 studies and compared the results to data extracted by human coding. Results indicate that LLMs struggled with extracting data accurately and therefore are not ready to be used as primary data extraction tools without explicit human validation of the data extracted. These findings highlight the dire need for a human-in-the-loop (HIL) approach to AI-assisted data extraction. We then propose a HIL workflow and introduce and describe the development of a free, web-based, open-source tool designed to support user-friendly, human-validated data extraction with LLMs.