# SESR-Eval: Dataset for Evaluating LLMs in the Title-Abstract Screening of Systematic Reviews

arXiv: https://arxiv.org/abs/2507.19027 · HTML: https://mlsanity.com/paper/2507.19027/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Jul 2025.

## Built on by (most cited first, 7 of 7)

- [LLM4SCREENLIT: Recommendations on Assessing the Performance of Large Language Models for Screening Literature in Systematic Reviews](https://mlsanity.com/paper/2511.12635/) (2025)
- [AISysRev -- LLM-based Tool for Title-abstract Screening](https://mlsanity.com/paper/2510.06708/) (2025)
- [A Reproducible Optimisation Protocol for Calibrating Prompt-Based Large Language Model Workflows in Evidence Synthesis](https://mlsanity.com/paper/2605.06937/) (2026)
- [Understanding LLMs in Title-Abstract Screening: From Disagreements to Recommendations](https://mlsanity.com/paper/2606.17588/) (2026)
- [Class Imbalance and Batch Effects in LLM-Based Screening for Systematic Reviews](https://mlsanity.com/paper/2608.14737/) (2026)
- [A Benchmark Framework for Screening Automation in Systematic Reviews](https://mlsanity.com/paper/2609.30298/) (2026)
- [Has LLM Screening Performance Stalled in Software Engineering Systematic Reviews?](https://mlsanity.com/paper/2610.10633/) (2026)

## Abstract (arXiv)

Background: The use of large language models (LLMs) in the title-abstract screening process of systematic reviews (SRs) has shown promising results, but suffers from limited performance evaluation. Aims: Create a benchmark dataset to evaluate the performance of LLMs in the title-abstract screening process of SRs. Provide evidence whether using LLMs in title-abstract screening in software engineering is advisable. Method: We start with 169 SR research artifacts and find 24 of those to be suitable for inclusion in the dataset. Using the dataset we benchmark title-abstract screening using 9 LLMs. Results: We present the SESR-Eval (Software Engineering Systematic Review Evaluation) dataset containing 34,528 labeled primary studies, sourced from 24 secondary studies published in software engineering (SE) journals. Most LLMs performed similarly and the differences in screening accuracy between secondary studies are greater than differences between LLMs. The cost of using an LLM is relatively low - less than 40 per secondary study even for the most expensive model. Conclusions: Our benchmark enables monitoring AI performance in the screening task of SRs in software engineering. At present, LLMs are not yet recommended for automating the title-abstract screening process, since accuracy varies widely across secondary studies, and no LLM managed a high recall with reasonable precision. In future, we plan to investigate factors that influence LLM screening performance between studies.