# Exploring the Impact of Instruction-Tuning on LLM's Susceptibility to Misinformation

arXiv: https://arxiv.org/abs/2507.18203 · HTML: https://mlsanity.com/paper/2507.18203/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security.
- Builds on 20 of its 29 references that are in the map.
- First posted on arXiv 24 Jul 2025.

## Placed in

- LLM safety and security / [Safety alignment, refusal and guardrails](https://mlsanity.com/safety/safety-alignment-refusal-and-guardrails/)

## Builds on (most cited first, 20 of 20)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://mlsanity.com/paper/1905.10044/) (2019)
- [LIMA: Less Is More for Alignment](https://mlsanity.com/paper/2305.11206/) (2023)
- [Discovering Language Model Behaviors with Model-Written Evaluations](https://mlsanity.com/paper/2212.09251/) (2022)
- [How Much Knowledge Can You Pack Into the Parameters of a Language Model?](https://mlsanity.com/paper/2002.08910/) (2020)
- [Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflicts](https://mlsanity.com/paper/2305.13300/) (2023)
- [The Earth is Flat because...: Investigating LLMs' Belief towards Misinformation via Persuasive Conversation](https://mlsanity.com/paper/2312.09085/) (2023)
- [Tug-of-War Between Knowledge: Exploring and Resolving Knowledge Conflicts in Retrieval-Augmented Language Models](https://mlsanity.com/paper/2402.14409/) (2024)
- [Evaluating the Instruction-Following Robustness of Large Language Models to Prompt Injection](https://mlsanity.com/paper/2308.10819/) (2023)
- [From Language Modeling to Instruction Following: Understanding the Behavior Shift in LLMs after Instruction Tuning](https://mlsanity.com/paper/2310.00492/) (2023)
- [Why So Gullible? Enhancing the Robustness of Retrieval-Augmented Models against Counterfactual Noise](https://mlsanity.com/paper/2305.01579/) (2023)
- [Attacking Open-domain Question Answering by Injecting Misinformation](https://mlsanity.com/paper/2110.07803/) (2021)
- [Merging Generated and Retrieved Knowledge for Open-Domain QA](https://mlsanity.com/paper/2310.14393/) (2023)
- [Intuitive or Dependent? Investigating LLMs' Behavior Style to Conflicting Prompts](https://mlsanity.com/paper/2309.17415/) (2023)
- [Preemptive Answer "Attacks" on Chain-of-Thought Reasoning](https://mlsanity.com/paper/2405.20902/) (2024)
- [Defending Against Disinformation Attacks in Open-Domain Question Answering](https://mlsanity.com/paper/2212.10002/) (2022)

## Built on by (most cited first, 4 of 4)

- [The Facade of Truth: Uncovering and Mitigating LLM Susceptibility to Deceptive Evidence](https://mlsanity.com/paper/2601.05478/) (2026)
- [MisSpans: Fine-Grained False Span Identification in Cross-Domain Fake News](https://mlsanity.com/paper/2601.04857/) (2026)
- [How Large Language Models Balance Internal Knowledge with User and Document Assertions](https://mlsanity.com/paper/2604.22193/) (2026)
- [Large Language Models in Misinformation Ecosystems: Misuse, Defense, and Vulnerability](https://mlsanity.com/paper/2607.10402/) (2026)

## Abstract (arXiv)

Instruction-tuning enhances the ability of large language models (LLMs) to follow user instructions more accurately, improving usability while reducing harmful outputs. However, this process may increase the model's dependence on user input, potentially leading to the unfiltered acceptance of misinformation and the generation of hallucinations. Existing studies primarily highlight that LLMs are receptive to external information that contradict their parametric knowledge, but little research has been conducted on the direct impact of instruction-tuning on this phenomenon. In our study, we investigate the impact of instruction-tuning on LLM's susceptibility to misinformation. Our analysis reveals that instruction-tuned LLMs are significantly more likely to accept misinformation when it is presented by the user. A comparison with base models shows that instruction-tuning increases reliance on user-provided information, shifting susceptibility from the assistant role to the user role. Furthermore, we explore additional factors influencing misinformation susceptibility, such as the role of the user in prompt structure, misinformation length, and the presence of warnings in the system prompt. Our findings underscore the need for systematic approaches to mitigate unintended consequences of instruction-tuning and enhance the reliability of LLMs in real-world applications.