# Guardrails for avoiding harmful medical product recommendations and off-label promotion in generative AI models

arXiv: https://arxiv.org/abs/2406.16455 · HTML: https://mlsanity.com/paper/2406.16455/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security.
- Builds on 22 of its 69 references that are in the map.
- First posted on arXiv 24 Jun 2024.

## Placed in

- LLM safety and security / [Safety alignment, refusal and guardrails](https://mlsanity.com/safety/safety-alignment-refusal-and-guardrails/)

## Builds on (most cited first, 22 of 22)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Retrieval-Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2312.10997/) (2023)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [A General Language Assistant as a Laboratory for Alignment](https://mlsanity.com/paper/2112.00861/) (2021)
- [Red Teaming Language Models with Language Models](https://mlsanity.com/paper/2202.03286/) (2022)
- [Discovering Language Model Behaviors with Model-Written Evaluations](https://mlsanity.com/paper/2212.09251/) (2022)
- [BERTopic: Neural topic modeling with a class-based TF-IDF procedure](https://mlsanity.com/paper/2203.05794/) (2022)
- [Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned](https://mlsanity.com/paper/2209.07858/) (2022)
- [Improving alignment of dialogue agents via targeted human judgements](https://mlsanity.com/paper/2209.14375/) (2022)
- [Med-BERT: pre-trained contextualized embeddings on large-scale structured electronic health records for disease prediction](https://mlsanity.com/paper/2005.12833/) (2020)
- [Towards Generalist Biomedical AI](https://mlsanity.com/paper/2307.14334/) (2023)
- [Adapted Large Language Models Can Outperform Medical Experts in Clinical Text Summarization](https://mlsanity.com/paper/2309.07430/) (2023)
- [The Capacity for Moral Self-Correction in Large Language Models](https://mlsanity.com/paper/2302.07459/) (2023)
- [Recipes for Safety in Open-domain Chatbots](https://mlsanity.com/paper/2010.07079/) (2020)
- [MiME: Multilevel Medical Embedding of Electronic Health Records for Predictive Healthcare](https://mlsanity.com/paper/1810.09593/) (2018)
- [Building Guardrails for Large Language Models](https://mlsanity.com/paper/2402.01822/) (2024)
- [AART: AI-Assisted Red-Teaming with Diverse Data Generation for New LLM-powered Applications](https://mlsanity.com/paper/2311.08592/) (2023)
- [Medical Concept Embedding with Time-Aware Attention](https://mlsanity.com/paper/1806.02873/) (2018)
- [Using Twitter Data to Understand Public Perceptions of Approved versus Off-label Use for COVID-19-related Medications](https://mlsanity.com/paper/2206.14358/) (2022)
- [What Does DALL-E 2 Know About Radiology?](https://mlsanity.com/paper/2209.13696/) (2022)
- [Development and Testing of a Novel Large Language Model-Based Clinical Decision Support Systems for Medication Safety in 12 Clinical Specialties](https://mlsanity.com/paper/2402.01741/) (2024)

## Built on by (most cited first, 1 of 1)

- [Building a Domain-specific Guardrail Model in Production](https://mlsanity.com/paper/2408.01452/) (2024)

## Abstract (arXiv)

Generative AI (GenAI) models have demonstrated remarkable capabilities in a wide variety of medical tasks. However, as these models are trained using generalist datasets with very limited human oversight, they can learn uses of medical products that have not been adequately evaluated for safety and efficacy, nor approved by regulatory agencies. Given the scale at which GenAI may reach users, unvetted recommendations pose a public health risk. In this work, we propose an approach to identify potentially harmful product recommendations, and demonstrate it using a recent multimodal large language model.