# An Audit and Analysis of LLM-Assisted Health Misinformation Jailbreaks Against LLMs

arXiv: https://arxiv.org/abs/2508.10010 · HTML: https://mlsanity.com/paper/2508.10010/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security.
- Builds on 13 of its 41 references that are in the map.
- First posted on arXiv 6 Aug 2025.

## Placed in

- LLM safety and security / [Safety alignment, refusal and guardrails](https://mlsanity.com/safety/safety-alignment-refusal-and-guardrails/)

## Builds on (most cited first, 13 of 13)

- [Ethical and social risks of harm from Language Models](https://mlsanity.com/paper/2112.04359/) (2021)
- [WildChat: 1M ChatGPT Interaction Logs in the Wild](https://mlsanity.com/paper/2405.01470/) (2024)
- [How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs](https://mlsanity.com/paper/2401.06373/) (2024)
- [From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge](https://mlsanity.com/paper/2411.16594/) (2024)
- [How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection](https://mlsanity.com/paper/2301.07597/) (2023)
- [Can LLM-Generated Misinformation Be Detected?](https://mlsanity.com/paper/2309.13788/) (2023)
- [Factuality Challenges in the Era of Large Language Models](https://mlsanity.com/paper/2310.05189/) (2023)
- [r/Fakeddit: A New Multimodal Benchmark Dataset for Fine-grained Fake News Detection](https://mlsanity.com/paper/1911.03854/) (2019)
- [Tricking LLMs into Disobedience: Formalizing, Analyzing, and Detecting Jailbreaks](https://mlsanity.com/paper/2305.14965/) (2023)
- [Exploring the Deceptive Power of LLM-Generated Fake News: A Study of Real-World Detection Challenges](https://mlsanity.com/paper/2403.18249/) (2024)
- [What Features in Prompts Jailbreak LLMs? Investigating the Mechanisms Behind Attacks](https://mlsanity.com/paper/2411.03343/) (2024)
- [Towards Safe AI Clinicians: A Comprehensive Study on Large Language Model Jailbreaking in Healthcare](https://mlsanity.com/paper/2501.18632/) (2025)
- [Hidden You Malicious Goal Into Benign Narratives: Jailbreak Large Language Models through Logic Chain Injection](https://mlsanity.com/paper/2404.04849/) (2024)

## Built on by (most cited first, 1 of 1)

- [To Lie or Not to Lie? Investigating The Biased Spread of Global Lies by LLMs](https://mlsanity.com/paper/2604.06552/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) are a double-edged sword capable of generating harmful misinformation -- inadvertently, or when prompted by "jailbreak" attacks that attempt to produce malicious outputs. LLMs could, with additional research, be used to detect and prevent the spread of misinformation. In this paper, we investigate the efficacy and characteristics of LLM-produced jailbreak attacks that cause other models to produce harmful medical misinformation. We also study how misinformation generated by jailbroken LLMs compares to typical misinformation found on social media, and how effectively it can be detected using standard machine learning approaches. Specifically, we closely examine 109 distinct attacks against three target LLMs and compare the attack prompts to in-the-wild health-related LLM queries. We also examine the resulting jailbreak responses, comparing the generated misinformation to health-related misinformation on Reddit. Our findings add more evidence that LLMs can be effectively used to detect misinformation from both other LLMs and from people, and support a body of work suggesting that with careful design, LLMs can contribute to a healthier overall information ecosystem.