mlsanityData to 7 Oct 2026

Wait, but Tylenol is Acetaminophen... Investigating and Improving Language Models' Ability to Resist Requests for Misinformation

arXiv 2409.20385 ↗First posted 30 Sep 2024cs.CLcited by 2

Not on the map. Why?

Abstract (arXiv)

Background: Large language models (LLMs) are trained to follow directions, but this introduces a vulnerability to blindly comply with user requests even if they generate wrong information. In medicine, this could accelerate the generation of misinformation that impacts human well-being. Objectives/Methods: We analyzed compliance to requests to generate misleading content about medications in settings where models know the request is illogical. We investigated whether in-context directions and instruction-tuning of LLMs to prioritize logical reasoning over compliance reduced misinformation risk. Results: While all frontier LLMs complied with misinformation requests, both prompt-based and parameter-based approaches can improve the detection of logic flaws in requests and prevent the dissemination of medical misinformation. Conclusion: Shifting LLMs to prioritize logic over compliance could reduce risks of exploitation for medical misinformation.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 31 of its 41 references that are in the map. First posted on arXiv 30 Sep 2024.

Builds on

31 of 41 references.

References · 31

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2017Deep reinforcement learning from human preferencescited by 3,846
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  8. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  9. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  10. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  11. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  12. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
19 more
  1. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  2. 2023Towards Understanding Sycophancy in Language Modelscited by 988
  3. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  4. 2022Galactica: A Large Language Model for Sciencecited by 736
  5. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 533
  6. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 475
  7. 2023Adapted Large Language Models Can Outperform Medical Experts in Clinical Text Summarizationcited by 244
  8. 2023A Wolf in Sheep's Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easilycited by 242
  9. 2024Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokenscited by 148
  10. 2023Physics of Language Models: Part 3.2, Knowledge Manipulationcited by 142
  11. 2023The Earth is Flat because...: Investigating LLMs' Belief towards Misinformation via Persuasive Conversationcited by 107
  12. 2024Making Them Ask and Answer: Jailbreaking Large Language Models in Few Queries via Disguise and Reconstructioncited by 95
  13. 2021Think you have Solved Direct-Answer Question Answering? Try ARC-DA, the Direct-Answer AI2 Reasoning Challengecited by 79
  14. 2023Cognitive Overload: Jailbreaking Large Language Models with Overloaded Logical Thinkingcited by 78
  15. 2023Large Language Models Help Humans Verify Truthfulness -- Except When They Are Convincingly Wrongcited by 51
  16. 2024Eraser: Jailbreaking Defense in Large Language Models via Unlearning Harmful Knowledgecited by 48
  17. 2024How do Large Language Models Navigate Conflicts between Honesty and Helpfulness?cited by 27
  18. 2023Evaluation of ChatGPT Family of Models for Biomedical Reasoning and Classificationcited by 17
  19. 2024Language Models are Surprisingly Fragile to Drug Names in Biomedical Benchmarkscited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025Retrieval-augmented systems can be dangerous medical communicatorscited by 5
  2. 2026MedRedFlag: Investigating how LLMs Redirect Misconceptions in Real-World Health Communicationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Susceptibility of Large Language Models to User-Driven Factors in Medical Queries
  2. 2025Towards Transparent Reasoning: What Drives Faithfulness in Large Language Models?
  3. 2025MediEval: A Unified Medical Benchmark for Patient-Contextual and Knowledge-Grounded Reasoning in LLMs
  4. 2026Measuring Epistemic Resilience of LLMs Under Misleading Medical Context
  5. 2023Minimizing Factual Inconsistency and Hallucination in Large Language Models
  6. 2026When Medical Safety Alignment Fails: A Benchmark for Evaluating LLMs on High-Risk Medical Queries
  7. 2026Rethinking LLM Verification: Evidence Structure, Uncertainty, and Selective Refinement
  8. 2026Towards Reliable Medical LLMs: Benchmarking and Enhancing Confidence Estimation of Large Language Models in Medical Consultation