mlsanityData to 7 Oct 2026

Guardrails for avoiding harmful medical product recommendations and off-label promotion in generative AI models

arXiv 2406.16455 ↗First posted 24 Jun 2024cs.AIcited by 1

On the map: LLM safety and security → Safety alignment, refusal and guardrails

Abstract (arXiv)

Generative AI (GenAI) models have demonstrated remarkable capabilities in a wide variety of medical tasks. However, as these models are trained using generalist datasets with very limited human oversight, they can learn uses of medical products that have not been adequately evaluated for safety and efficacy, nor approved by regulatory agencies. Given the scale at which GenAI may reach users, unvetted recommendations pose a public health risk. In this work, we propose an approach to identify potentially harmful product recommendations, and demonstrate it using a recent multimodal large language model.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security. Builds on 22 of its 69 references that are in the map. First posted on arXiv 24 Jun 2024.

Builds on

22 of 69 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Building Guardrails for Large Language Models2 Feb
  2. 2023The Capacity for Moral Self-Correction in Large Language Models15 Feb

Other references · 20

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  3. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  4. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  5. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  6. 2022Red Teaming Language Models with Language Modelscited by 983
  7. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  8. 2022BERTopic: Neural topic modeling with a class-based TF-IDF procedurecited by 779
  9. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  10. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  11. 2020Med-BERT: pre-trained contextualized embeddings on large-scale structured electronic health records for disease predictioncited by 368
  12. 2023Towards Generalist Biomedical AIcited by 286
8 more
  1. 2023Adapted Large Language Models Can Outperform Medical Experts in Clinical Text Summarizationcited by 244
  2. 2020Recipes for Safety in Open-domain Chatbotscited by 168
  3. 2018MiME: Multilevel Medical Embedding of Electronic Health Records for Predictive Healthcarecited by 72
  4. 2023AART: AI-Assisted Red-Teaming with Diverse Data Generation for New LLM-powered Applicationscited by 53
  5. 2018Medical Concept Embedding with Time-Aware Attentioncited by 11
  6. 2022Using Twitter Data to Understand Public Perceptions of Approved versus Off-label Use for COVID-19-related Medicationscited by 8
  7. 2022What Does DALL-E 2 Know About Radiology?cited by 7
  8. 2024Development and Testing of a Novel Large Language Model-Based Clinical Decision Support Systems for Medication Safety in 12 Clinical Specialtiescited by 5

Built on by

Cited by year

Citing papers per year: 2024: 1.

1 paper.

  1. 2024Building a Domain-specific Guardrail Model in Productioncited by 6

Similar, not cited

Close in wording, no citation either way

  1. 2025A Systematic Analysis of Declining Medical Safety Messaging in Generative AI Models
  2. 2024The Potential and Perils of Generative Artificial Intelligence for Quality Improvement and Patient Safety
  3. 2025Generative AI for Healthcare: Fundamentals, Challenges, and Perspectives
  4. 2024Ensuring Safety and Trust: Analyzing the Risks of Large Language Models in Medicine
  5. 2025Computational Safety for Generative AI: A Hypothesis Testing Perspective
  6. 2026CareGuardAI: Context-Aware Multi-Agent Guardrails for Clinical Safety & Hallucination Mitigation in Patient-Facing LLMs
  7. 2026Improving Rare Medication Recommendation with Counterfactual Data Augmentation and Large Language Models
  8. 2024AI Safety in Generative AI Large Language Models: A Survey