mlsanityData to 14 Oct 2026

Auxiliary uncertainty signals for LLM-assisted systematic review screening: a benchmark across eight Cohen drug-class reviews

arXiv 2608.14551 ↗First posted 3 May 2026cs.CL cs.DL cs.IRcited by 0

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) are increasingly used for title-abstract screening in systematic reviews, but their decisions lack calibrated uncertainty. We show that an auxiliary BERT+GCN classifier supplies a structured uncertainty signal that improves LLM screening efficiency, and we identify the prompt-delivery strategy that maximises the benefit-to-cost ratio. We evaluate five LLM prompt-delivery conditions on eight drug-class datasets from the Cohen (2006) benchmark using 3 seeds x 5-fold stratified cross-validation (600 fold-level results). A BERT+GCN model trained per fold classifies each test paper as INCLUDE, EXCLUDE, or MAYBE via two spectral tests (algebraic radical and categorical paradox). Conditions vary information content (none / label / full scores), selectivity (all papers vs. MAYBE only), and timing (proactive vs. reactive two-pass). A cross-model pilot against gpt-4.1-mini on three datasets tests cross-generation transfer. Three findings: (i) Full-context delivery yields significant gains in F1 (+0.011, paired Wilcoxon p=0.008) and WSS@95 (+0.050, p=0.039) at a 1.28x token-cost premium, while preserving recall. (ii) MAYBE-only routing is Pareto-optimal: highest mean recall (0.92) and AUC-ROC (0.54) at only 1.05x baseline cost -- one sixth of full-context overhead. (iii) The two-pass design escalates 22.2% +/- 8.8% of records yet never revises its decision (0% flip rate across all datasets and folds), giving decisive evidence that current instruction-tuned LLMs cannot self-triage. The cross-model pilot shows an identical +0.8% recall uplift for both LLM generations. A per-paper ablation across 20,796 observations shows the dual paradox test reduces empirically to a one-line logit-gap criterion. We release the full pipeline; the 600-run experiment replays in under one hour from cached LLM responses.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 17 of its 28 references that are in the map. First posted on arXiv 3 May 2026.

Builds on

17 of 28 references.

References · 17

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,361
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  4. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  5. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  6. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  7. 2017Modeling Relational Data with Graph Convolutional Networkscited by 1,694
  8. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 1,009
  9. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 974
  10. 2018Graph Convolutional Networks for Text Classificationcited by 345
  11. 2023Tool Learning with Foundation Modelscited by 302
  12. 2022OpenAlex: A fully-open index of scholarly works, authors, venues, institutions, and conceptscited by 177
5 more
  1. 2021BertGCN: Transductive Text Classification by Combining GCN and BERTcited by 74
  2. 2023Can large language models replace humans in the systematic review process? Evaluating GPT-4's efficacy in screening and extracting data from peer-reviewed and grey literature in multiple languagescited by 36
  3. 2020Open Source Software for Efficient and Transparent Reviewscited by 35
  4. 2023Automated Paper Screening for Clinical Reviews Using Large Language Modelscited by 29
  5. 2024Zero-shot Generative Large Language Models for Systematic Review Screening Automationcited by 14

Built on by

Not cited yet.