mlsanityData to 14 Oct 2026

Appraising the Potential Uses and Harms of LLMs for Medical Systematic Reviews

arXiv 2305.11828 ↗First posted 19 May 2023cs.CL cs.AI cs.HCcited by 18

Not on the map. Why?

Abstract (arXiv)

Medical systematic reviews play a vital role in healthcare decision making and policy. However, their production is time-consuming, limiting the availability of high-quality and up-to-date evidence summaries. Recent advancements in large language models (LLMs) offer the potential to automatically generate literature reviews on demand, addressing this issue. However, LLMs sometimes generate inaccurate (and potentially misleading) texts by hallucination or omission. In healthcare, this can make LLMs unusable at best and dangerous at worst. We conducted 16 interviews with international systematic review experts to characterize the perceived utility and risks of LLMs in the specific context of medical evidence reviews. Experts indicated that LLMs can assist in the writing process by drafting summaries, generating templates, distilling information, and crosschecking information. They also raised concerns regarding confidently composed but inaccurate LLM outputs and other potential downstream harms, including decreased accountability and proliferation of low-quality reviews. Informed by this qualitative analysis, we identify criteria for rigorous evaluation of biomedical LLMs aligned with domain expert views.

Key facts. Cited by 18 papers in the map (data through 14 Oct 2026). Builds on 22 of its 97 references that are in the map. First posted on arXiv 19 May 2023.

Builds on

22 of 97 references.

References · 22

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  5. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,947
  6. 2022Large Language Models Encode Clinical Knowledgecited by 1,830
  7. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  8. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,414
  9. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  10. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,135
  11. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 1,009
  12. 2022Galactica: A Large Language Model for Sciencecited by 744
10 more
  1. 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 606
  2. 2022BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Miningcited by 538
  3. 2019ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmissioncited by 493
  4. 2022News Summarization and Evaluation in the Era of GPT-3cited by 327
  5. 2019What Clinicians Want: Contextualizing Explainable Machine Learning for Clinical End Usecited by 135
  6. 2023Do We Still Need Clinical Language Models?cited by 98
  7. 2021ClimateBert: A Pretrained Language Model for Climate-Related Textcited by 90
  8. 2021MS2: Multi-Document Summarization of Medical Studiescited by 80
  9. 2020Generating (Factual?) Narrative Summaries of RCTs: Experiments with Neural Multi-Document Summarizationcited by 52
  10. 2022Characteristics of Harmful Text: Towards Rigorous Benchmarking of Language Modelscited by 49

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 7; 2025: 7; 2026: 2.

18 papers.

Sort:

By area:

  1. 2024BioMedLM: A 2.7B Parameter Language Model Trained On Biomedical Textcited by 86
  2. 2025Multi-Agent-as-Judge: Aligning LLM-Agent-Based Automated Evaluation with Multi-Dimensional Human Evaluationcited by 30
  3. 2024CHIME: LLM-Assisted Hierarchical Organization of Scientific Studies for Literature Review Supportcited by 29
  4. 2024The emergence of Large Language Models (LLM) as a tool in literature reviews: an LLM automated systematic reviewcited by 21
  5. 2024Accelerating Clinical Evidence Synthesis with Large Language Modelscited by 16
  6. 2025How Far Are AI Scientists from Changing the World?cited by 15
  7. 2025A foundation model for human-AI collaboration in medical literature miningcited by 11
  8. 2024A Perspective for Adapting Generalist AI to Specialized Medical AI Applications and Their Challengescited by 8
  9. 2025Can LLMs Help Uncover Insights about LLMs? A Large-Scale, Evolving Literature Analysis of Frontier LLMscited by 3
  10. 2025Query-driven Document-level Scientific Evidence Extraction from Biomedical Studiescited by 3
  11. 2026Faithfulness vs. Safety: Evaluating LLM Behavior Under Counterfactual Medical Evidencecited by 3
  12. 2023CSMeD: Bridging the Dataset Gap in Automated Citation Screening for Systematic Literature Reviewscited by 2
6 more
  1. 2023Keeping Users Engaged During Repeated Administration of the Same Questionnaire: Using Large Language Models to Reliably Diversify Questionscited by 0
  2. 2024Are LLMs Effective Backbones for Fine-tuning? An Experimental Investigation of Supervised LLMs on Chinese Short Text Matchingcited by 0
  3. 2024RoBIn: A Transformer-Based Model For Risk Of Bias Inference With Machine Reading Comprehensioncited by 0
  4. 2025Accelerating Antibiotic Discovery with Large Language Models and Knowledge Graphscited by 0
  5. 2025CareMedEval dataset: Evaluating Critical Appraisal and Reasoning in the Biomedical Fieldcited by 0
  6. 2026Charting the Growth of Social-Physical HRI (spHRI): A Systematic Review Pipeline Augmented by Small Language Modelscited by 0