# Automated Paper Screening for Clinical Reviews Using Large Language Models

arXiv: https://arxiv.org/abs/2305.00844 · HTML: https://mlsanity.com/paper/2305.00844/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 2 of its 42 references that are in the map.
- First posted on arXiv 1 May 2023.

## Builds on (most cited first, 2 of 2)

- [Open Source Software for Efficient and Transparent Reviews](https://mlsanity.com/paper/2006.12166/) (2020)
- [Trialstreamer: Mapping and Browsing Medical Evidence in Real-Time](https://mlsanity.com/paper/2005.10865/) (2020)

## Built on by (most cited first, 29 of 29)

- [Large language models for automated scholarly paper review: A survey](https://mlsanity.com/paper/2501.10326/) (2025)
- [Can large language models replace humans in the systematic review process? Evaluating GPT-4's efficacy in screening and extracting data from peer-reviewed and grey literature in multiple languages](https://mlsanity.com/paper/2310.17526/) (2023)
- [The emergence of Large Language Models (LLM) as a tool in literature reviews: an LLM automated systematic review](https://mlsanity.com/paper/2409.04600/) (2024)
- [A Comprehensive Survey on Evaluating Large Language Model Applications in the Medical Industry](https://mlsanity.com/paper/2404.15777/) (2024)
- [The Promise and Challenges of Using LLMs to Accelerate the Screening Process of Systematic Reviews](https://mlsanity.com/paper/2404.15667/) (2024)
- [Toward Automated Simulation Research Workflow through LLM Prompt Engineering Design](https://mlsanity.com/paper/2408.15512/) (2024)
- [Bio-SIEVE: Exploring Instruction Tuning Large Language Models for Systematic Review Automation](https://mlsanity.com/paper/2308.06610/) (2023)
- [RelevAI-Reviewer: A Benchmark on AI Reviewers for Survey Paper Relevance](https://mlsanity.com/paper/2406.10294/) (2024)
- [Automated title and abstract screening for scoping reviews using the GPT-4 Large Language Model](https://mlsanity.com/paper/2311.07918/) (2023)
- [LLM4SCREENLIT: Recommendations on Assessing the Performance of Large Language Models for Screening Literature in Systematic Reviews](https://mlsanity.com/paper/2511.12635/) (2025)
- [Measuring Risk of Bias in Biomedical Reports: The RoBBR Benchmark](https://mlsanity.com/paper/2411.18831/) (2024)
- [Streamlining Systematic Reviews: A Novel Application of Large Language Models](https://mlsanity.com/paper/2412.15247/) (2024)
- [Large Language Models for Healthcare Text Classification: A Systematic Review](https://mlsanity.com/paper/2503.01159/) (2025)
- [Evaluating Prompting Strategies and Large Language Models in Systematic Literature Review Screening: Relevance and Task-Stage Classification](https://mlsanity.com/paper/2510.16091/) (2025)
- [AutoSynthesis: An agentic system for automated meta-analysis](https://mlsanity.com/paper/2607.15247/) (2026)
- [Jev in Medicine: A Benchmark Evaluation](https://mlsanity.com/paper/2609.34024/) (2026)
- [Generative AI in Medicine](https://mlsanity.com/paper/2412.10337/) (2024)
- [Large language models streamline automated systematic review: A preliminary study](https://mlsanity.com/paper/2502.15702/) (2025)
- [Assisting humans in complex comparisons: automated information comparison at scale](https://mlsanity.com/paper/2404.04351/) (2024)
- [Enhancing LLMs for Identifying and Prioritizing Important Medical Jargons from Electronic Health Record Notes Utilizing Data Augmentation: A Comparative Study](https://mlsanity.com/paper/2502.16022/) (2025)
- [CKD-EHR:Clinical Knowledge Distillation for Electronic Health Records](https://mlsanity.com/paper/2506.15118/) (2025)
- [An Auditable Pipeline for Fuzzy Full-Text Screening in Systematic Reviews: Integrating Contrastive Semantic Highlighting and LLM Judgment](https://mlsanity.com/paper/2508.15822/) (2025)
- [Leveraging LLMs for Title and Abstract Screening for Systematic Review: A Cost-Effective Dynamic Few-Shot Learning Approach](https://mlsanity.com/paper/2512.11261/) (2025)
- [Improving Behavioral Alignment in LLM Social Simulations via Context Formation and Navigation](https://mlsanity.com/paper/2601.01546/) (2026)
- [TiAb Review Plugin: A Browser-Based Tool for AI-Assisted Study Selection in Systematic Reviews](https://mlsanity.com/paper/2604.08602/) (2026)
- [A Tertiary Review of Large Language Model-Based Code Generating Tasks: Trends, Challenges, and Future Directions](https://mlsanity.com/paper/2605.25536/) (2026)
- [Evaluation design conditions the expert-vs-auto MeSH gap: a controlled comparison of bag-of-words and BiomedBERT on the Cohen benchmark](https://mlsanity.com/paper/2607.21685/) (2026)
- [Auxiliary uncertainty signals for LLM-assisted systematic review screening: a benchmark across eight Cohen drug-class reviews](https://mlsanity.com/paper/2608.14551/) (2026)
- [Using Human-LLM Disagreement to Improve Checklist-Based Quality Appraisal](https://mlsanity.com/paper/2608.20385/) (2026)

## Abstract (arXiv)

Objective: To assess the performance of the OpenAI GPT API in accurately and efficiently identifying relevant titles and abstracts from real-world clinical review datasets and compare its performance against ground truth labelling by two independent human reviewers. Methods: We introduce a novel workflow using the OpenAI GPT API for screening titles and abstracts in clinical reviews. A Python script was created to make calls to the GPT API with the screening criteria in natural language and a corpus of title and abstract datasets that have been filtered by a minimum of two human reviewers. We compared the performance of our model against human-reviewed papers across six review papers, screening over 24,000 titles and abstracts. Results: Our results show an accuracy of 0.91, a sensitivity of excluded papers of 0.91, and a sensitivity of included papers of 0.76. On a randomly selected subset of papers, the GPT API demonstrated the ability to provide reasoning for its decisions and corrected its initial decision upon being asked to explain its reasoning for a subset of incorrect classifications. Conclusion: The GPT API has the potential to streamline the clinical review process, save valuable time and effort for researchers, and contribute to the overall quality of clinical reviews. By prioritizing the workflow and acting as an aid rather than a replacement for researchers and reviewers, the GPT API can enhance efficiency and lead to more accurate and reliable conclusions in medical research.