mlsanityData to 7 Oct 2026

Class Imbalance and Batch Effects in LLM-Based Screening for Systematic Reviews

arXiv 2608.14737 ↗First posted 13 Aug 2026cs.CL cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

This study analyses LLMs in imbalanced binary classification, using study screening in systematic reviews as the application domain. An experiment was conducted in five reviews, comparing individual and batch processing, with and without prevalence metadata. The results indicate a limited influence of the prevalence metadata, with no evidence that it improves performance. In contrast, batch processing produced larger behavioral changes that varied according to the prevalence of the class. The aggregate and item-level analyses did not always coincide. Therefore, batch processing should be evaluated not only in terms of cost, but also in relation to its effects on decision-making behavior.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 14 of its 24 references that are in the map. First posted on arXiv 13 Aug 2026.

Builds on

14 of 24 references.

References · 14

  1. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  2. 2023Batch Prompting: Efficient Inference with Large Language Model APIscited by 84
  3. 2020Growth rates of modern science: A latent piecewise growth curve approach to model publication numbers from established and new literature databasescited by 75
  4. 2023Are You Sure? Challenging LLMs Leads to Performance Drops in The FlipFlop Experimentcited by 71
  5. 2021How to Certify Machine Learning Based Safety-critical Systems? A Systematic Literature Reviewcited by 36
  6. 2024Zero-shot Generative Large Language Models for Systematic Review Screening Automationcited by 14
  7. 2024The Promise and Challenges of Using LLMs to Accelerate the Screening Process of Systematic Reviewscited by 14
  8. 2019Time Pressure in Software Engineering: A Systematic Reviewcited by 8
  9. 2025SESR-Eval: Dataset for Evaluating LLMs in the Title-Abstract Screening of Systematic Reviewscited by 7
  10. 2022A Decade of Code Comment Quality Assessment: A Systematic Literature Reviewcited by 5
  11. 2025LLM4SCREENLIT: Recommendations on Assessing the Performance of Large Language Models for Screening Literature in Systematic Reviewscited by 5
  12. 2022Please, Don't Forget the Difference and the Confidence Interval when Seeking for the State-of-the-Art Statuscited by 3
2 more
  1. 2026Researchers waste 80% of LLM annotation costs by classifying one text at a timecited by 1
  2. 2026Beyond Accuracy: LLM Variability in Evidence Screening for Software Engineering SLRscited by 1

Built on by

Not cited yet.