# Class Imbalance and Batch Effects in LLM-Based Screening for Systematic Reviews

arXiv: https://arxiv.org/abs/2608.14737 · HTML: https://mlsanity.com/paper/2608.14737/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 24 references that are in the map.
- First posted on arXiv 13 Aug 2026.

## Builds on (most cited first, 14 of 14)

- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Batch Prompting: Efficient Inference with Large Language Model APIs](https://mlsanity.com/paper/2301.08721/) (2023)
- [Are You Sure? Challenging LLMs Leads to Performance Drops in The FlipFlop Experiment](https://mlsanity.com/paper/2311.08596/) (2023)
- [Growth rates of modern science: A latent piecewise growth curve approach to model publication numbers from established and new literature databases](https://mlsanity.com/paper/2012.07675/) (2020)
- [How to Certify Machine Learning Based Safety-critical Systems? A Systematic Literature Review](https://mlsanity.com/paper/2107.12045/) (2021)
- [Zero-shot Generative Large Language Models for Systematic Review Screening Automation](https://mlsanity.com/paper/2401.06320/) (2024)
- [The Promise and Challenges of Using LLMs to Accelerate the Screening Process of Systematic Reviews](https://mlsanity.com/paper/2404.15667/) (2024)
- [Time Pressure in Software Engineering: A Systematic Review](https://mlsanity.com/paper/1901.05771/) (2019)
- [SESR-Eval: Dataset for Evaluating LLMs in the Title-Abstract Screening of Systematic Reviews](https://mlsanity.com/paper/2507.19027/) (2025)
- [A Decade of Code Comment Quality Assessment: A Systematic Literature Review](https://mlsanity.com/paper/2209.08165/) (2022)
- [LLM4SCREENLIT: Recommendations on Assessing the Performance of Large Language Models for Screening Literature in Systematic Reviews](https://mlsanity.com/paper/2511.12635/) (2025)
- [Please, Don't Forget the Difference and the Confidence Interval when Seeking for the State-of-the-Art Status](https://mlsanity.com/paper/2205.11134/) (2022)
- [Researchers waste 80% of LLM annotation costs by classifying one text at a time](https://mlsanity.com/paper/2604.03684/) (2026)
- [Beyond Accuracy: LLM Variability in Evidence Screening for Software Engineering SLRs](https://mlsanity.com/paper/2604.27006/) (2026)

## Abstract (arXiv)

This study analyses LLMs in imbalanced binary classification, using study screening in systematic reviews as the application domain. An experiment was conducted in five reviews, comparing individual and batch processing, with and without prevalence metadata. The results indicate a limited influence of the prevalence metadata, with no evidence that it improves performance. In contrast, batch processing produced larger behavioral changes that varied according to the prevalence of the class. The aggregate and item-level analyses did not always coincide. Therefore, batch processing should be evaluated not only in terms of cost, but also in relation to its effects on decision-making behavior.