# The emergence of Large Language Models (LLM) as a tool in literature reviews: an LLM automated systematic review

arXiv: https://arxiv.org/abs/2409.04600 · HTML: https://mlsanity.com/paper/2409.04600/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 203 references that are in the map.
- First posted on arXiv 6 Sep 2024.

## Builds on (most cited first, 30 of 35)

- [FacTool: Factuality Detection in Generative AI -- A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarios](https://mlsanity.com/paper/2307.13528/) (2023)
- [Concept Induction: Analyzing Unstructured Text with High-Level Concepts Using LLooM](https://mlsanity.com/paper/2404.12259/) (2024)
- [LitLLM: A Toolkit for Scientific Literature Review](https://mlsanity.com/paper/2402.01788/) (2024)
- [Can ChatGPT Write a Good Boolean Query for Systematic Review Literature Search?](https://mlsanity.com/paper/2302.03495/) (2023)
- [The Ethics of ChatGPT in Medicine and Healthcare: A Systematic Review on Large Language Models (LLMs)](https://mlsanity.com/paper/2403.14473/) (2024)
- [Can large language models replace humans in the systematic review process? Evaluating GPT-4's efficacy in screening and extracting data from peer-reviewed and grey literature in multiple languages](https://mlsanity.com/paper/2310.17526/) (2023)
- [Detecting LLM-Generated Text in Computing Education: A Comparative Study for ChatGPT Cases](https://mlsanity.com/paper/2307.07411/) (2023)
- [Automating Research Synthesis with Domain-Specific Large Language Model Fine-Tuning](https://mlsanity.com/paper/2404.08680/) (2024)
- [Automated Paper Screening for Clinical Reviews Using Large Language Models](https://mlsanity.com/paper/2305.00844/) (2023)
- [Clinfo.ai: An Open-Source Retrieval-Augmented Large Language Model System for Answering Medical Questions using Scientific Literature](https://mlsanity.com/paper/2310.16146/) (2023)
- [Appraising the Potential Uses and Harms of LLMs for Medical Systematic Reviews](https://mlsanity.com/paper/2305.11828/) (2023)
- [Zero-shot Generative Large Language Models for Systematic Review Screening Automation](https://mlsanity.com/paper/2401.06320/) (2024)
- [The Promise and Challenges of Using LLMs to Accelerate the Screening Process of Systematic Reviews](https://mlsanity.com/paper/2404.15667/) (2024)
- [Hierarchical Catalogue Generation for Literature Review: A Benchmark](https://mlsanity.com/paper/2304.03512/) (2023)
- [Assessing the Ability of ChatGPT to Screen Articles for Systematic Reviews](https://mlsanity.com/paper/2307.06464/) (2023)
- [Large Language Model Evaluation Via Multi AI Agents: Preliminary results](https://mlsanity.com/paper/2404.01023/) (2024)
- [Neural Rankers for Effective Screening Prioritisation in Medical Systematic Review Literature Search](https://mlsanity.com/paper/2212.09017/) (2022)
- [Exploring the use of a Large Language Model for data extraction in systematic reviews: a rapid feasibility study](https://mlsanity.com/paper/2405.14445/) (2024)
- [Bio-SIEVE: Exploring Instruction Tuning Large Language Models for Systematic Review Automation](https://mlsanity.com/paper/2308.06610/) (2023)
- [Generating Natural Language Queries for More Effective Systematic Review Screening Prioritisation](https://mlsanity.com/paper/2309.05238/) (2023)
- [Automated tabulation of clinical trial results: A joint entity and relation extraction approach with transformer-based language representations](https://mlsanity.com/paper/2112.05596/) (2021)
- [Open Tracing Tools: Overview and Critical Comparison](https://mlsanity.com/paper/2207.06875/) (2022)
- [GEAR-Up: Generative AI and External Knowledge-based Retrieval Upgrading Scholarly Article Searches for Systematic Reviews](https://mlsanity.com/paper/2312.09948/) (2023)
- [Streamlining the Selection Phase of Systematic Literature Reviews (SLRs) Using AI-Enabled GPT-4 Assistant API](https://mlsanity.com/paper/2402.18582/) (2024)
- [A Semi-Automated Solution Approach Recommender for a Given Use Case: a Case Study for AI/ML in Oncology via Scopus and OpenAI](https://mlsanity.com/paper/2307.04573/) (2023)
- [Preparing to Integrate Generative Pretrained Transformer Series 4 models into Genetic Variant Assessment Workflows: Assessing Performance, Drift, and Nondeterminism Characteristics Relative to Classifying Functional Evidence in Literature](https://mlsanity.com/paper/2312.13521/) (2023)
- [LLMs in HCI Data Work: Bridging the Gap Between Information Retrieval and Responsible Research Practices](https://mlsanity.com/paper/2403.18173/) (2024)
- [Systematic Review on Healthcare Systems Engineering utilizing ChatGPT](https://mlsanity.com/paper/2405.11817/) (2024)
- [Cascade Neural Ensemble for Identifying Scientifically Sound Articles](https://mlsanity.com/paper/2004.06222/) (2020)
- [Automatic Aspect Extraction from Scientific Texts](https://mlsanity.com/paper/2310.04074/) (2023)

## Built on by (most cited first, 21 of 21)

- [A Comprehensive Survey on Multi-Agent Cooperative Decision-Making: Scenarios, Approaches, Challenges and Perspectives](https://mlsanity.com/paper/2503.13415/) (2025)
- [Probing the Critical Point (CritPt) of AI Reasoning: a Frontier Physics Research Benchmark](https://mlsanity.com/paper/2509.26574/) (2025)
- [Knowledge Augmented Complex Problem Solving with Large Language Models: A Survey](https://mlsanity.com/paper/2505.03418/) (2025)
- [LLM4SCREENLIT: Recommendations on Assessing the Performance of Large Language Models for Screening Literature in Systematic Reviews](https://mlsanity.com/paper/2511.12635/) (2025)
- [Leveraging Large Language Models for Generating Research Topic Ontologies: A Multi-Disciplinary Study](https://mlsanity.com/paper/2508.20693/) (2025)
- [Leveraging LLMs for Semi-Automatic Corpus Filtration in Systematic Literature Reviews](https://mlsanity.com/paper/2510.11409/) (2025)
- [What Are We Measuring in NLG? A Meta-Analysis of Evaluation Trends 2020-2025](https://mlsanity.com/paper/2601.07648/) (2026)
- [SciHorizon-GENE: Benchmarking LLM for Life Sciences Inference from Gene Knowledge to Functional Understanding](https://mlsanity.com/paper/2601.12805/) (2026)
- [Mapping the Landscape of Artificial Intelligence in Life Cycle Assessment Using Large Language Models](https://mlsanity.com/paper/2602.22500/) (2026)
- [What do Large Language Models know about materials?](https://mlsanity.com/paper/2507.14586/) (2025)
- [An Auditable Pipeline for Fuzzy Full-Text Screening in Systematic Reviews: Integrating Contrastive Semantic Highlighting and LLM Judgment](https://mlsanity.com/paper/2508.15822/) (2025)
- [Multi-Dimensional Knowledge Profiling with Large-Scale Literature Database and Hierarchical Retrieval](https://mlsanity.com/paper/2601.15170/) (2026)
- [Evaluating Large Language Models for Abstract Evaluation Tasks: An Empirical Study](https://mlsanity.com/paper/2601.19925/) (2026)
- [FlyAOC: Evaluating Agentic Ontology Curation of Drosophila Scientific Knowledge Bases](https://mlsanity.com/paper/2602.09163/) (2026)
- [Attribution Bias in Large Language Models](https://mlsanity.com/paper/2604.05224/) (2026)
- [Agentic AI in Engineering and Manufacturing: Industry Perspectives on Utility, Adoption, Challenges, and Opportunities](https://mlsanity.com/paper/2604.09633/) (2026)
- [CheckSupport: A Local LLM-Powered Tool for Automated Manuscript Submission Checklist Selection and Completion](https://mlsanity.com/paper/2605.16377/) (2026)
- [GIScholarBench: Benchmarking LLM Overconfidence in GIS Research](https://mlsanity.com/paper/2606.08036/) (2026)
- [PRISMA-LLM: An Empirical Reporting Framework for AI-Assisted Systematic Reviews](https://mlsanity.com/paper/2609.11559/) (2026)
- [A Benchmark Framework for Screening Automation in Systematic Reviews](https://mlsanity.com/paper/2609.30298/) (2026)
- [Errors of LLM-Assisted Literature Retrieval in Environmental Science: A Comparison Study of Abstract versus Full-text Based Prompts](https://mlsanity.com/paper/2610.05690/) (2026)

## Abstract (arXiv)

Objective: This study aims to summarize the usage of Large Language Models (LLMs) in the process of creating a scientific review. We look at the range of stages in a review that can be automated and assess the current state-of-the-art research projects in the field. Materials and Methods: The search was conducted in June 2024 in PubMed, Scopus, Dimensions, and Google Scholar databases by human reviewers. Screening and extraction process took place in Covidence with the help of LLM add-on which uses OpenAI gpt-4o model. ChatGPT was used to clean extracted data and generate code for figures in this manuscript, ChatGPT and Scite.ai were used in drafting all components of the manuscript, except the methods and discussion sections. Results: 3,788 articles were retrieved, and 172 studies were deemed eligible for the final review. ChatGPT and GPT-based LLM emerged as the most dominant architecture for review automation (n=126, 73.2%). A significant number of review automation projects were found, but only a limited number of papers (n=26, 15.1%) were actual reviews that used LLM during their creation. Most citations focused on automation of a particular stage of review, such as Searching for publications (n=60, 34.9%), and Data extraction (n=54, 31.4%). When comparing pooled performance of GPT-based and BERT-based models, the former were better in data extraction with mean precision 83.0% (SD=10.4), and recall 86.0% (SD=9.8), while being slightly less accurate in title and abstract screening stage (Maccuracy=77.3%, SD=13.0). Discussion/Conclusion: Our LLM-assisted systematic review revealed a significant number of research projects related to review automation using LLMs. The results looked promising, and we anticipate that LLMs will change in the near future the way the scientific reviews are conducted.