# Can Large Language Models Provide Security & Privacy Advice? Measuring the Ability of LLMs to Refute Misconceptions

arXiv: https://arxiv.org/abs/2310.02431 · HTML: https://mlsanity.com/paper/2310.02431/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 62 references that are in the map.
- First posted on arXiv 3 Oct 2023.

## Builds on (most cited first, 28 of 28)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models](https://mlsanity.com/paper/2009.11462/) (2020)
- [LaMDA: Language Models for Dialog Applications](https://mlsanity.com/paper/2201.08239/) (2022)
- [A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity](https://mlsanity.com/paper/2302.04023/) (2023)
- [DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation](https://mlsanity.com/paper/1911.00536/) (2019)
- [Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models](https://mlsanity.com/paper/2309.01219/) (2023)
- [BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage](https://mlsanity.com/paper/2208.03188/) (2022)
- [Investigating the Factual Knowledge Boundary of Large Language Models with Retrieval Augmentation](https://mlsanity.com/paper/2307.11019/) (2023)
- [Hallucinations in Large Multilingual Translation Models](https://mlsanity.com/paper/2303.16104/) (2023)
- [Examining Zero-Shot Vulnerability Repair with Large Language Models](https://mlsanity.com/paper/2112.02125/) (2021)
- [PentestGPT: An LLM-empowered Automatic Penetration Testing Tool](https://mlsanity.com/paper/2308.06782/) (2023)
- [Stance Detection on Social Media: State of the Art and Trends](https://mlsanity.com/paper/2006.03644/) (2020)
- [Identifying and Mitigating the Security Risks of Generative AI](https://mlsanity.com/paper/2308.14840/) (2023)
- [How would Stance Detection Techniques Evolve after the Launch of ChatGPT?](https://mlsanity.com/paper/2212.14548/) (2022)
- [GODEL: Large-Scale Pre-Training for Goal-Directed Dialog](https://mlsanity.com/paper/2206.11309/) (2022)
- [In ChatGPT We Trust? Measuring and Characterizing the Reliability of ChatGPT](https://mlsanity.com/paper/2304.08979/) (2023)
- [Why So Toxic? Measuring and Triggering Toxic Behavior in Open-Domain Chatbots](https://mlsanity.com/paper/2209.03463/) (2022)
- [Dr ChatGPT, tell me what I want to hear: How prompt knowledge impacts health answer correctness](https://mlsanity.com/paper/2302.13793/) (2023)
- [Can Large Language Models Find And Fix Vulnerable Software?](https://mlsanity.com/paper/2308.10345/) (2023)
- [DIVAS: An LLM-based End-to-End Framework for SoC Security Analysis and Policy-based Protection](https://mlsanity.com/paper/2308.06932/) (2023)
- [ZeroLeak: Using LLMs for Scalable and Cost Effective Side-Channel Patching](https://mlsanity.com/paper/2308.13062/) (2023)

## Built on by (most cited first, 20 of 20)

- [A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly](https://mlsanity.com/paper/2312.02003/) (2023)
- [A Survey of Large Language Models in Medicine: Progress, Application, and Challenge](https://mlsanity.com/paper/2311.05112/) (2023)
- [Large Language Models for Cyber Security: A Systematic Literature Review](https://mlsanity.com/paper/2405.04760/) (2024)
- [Transformers and Large Language Models for Efficient Intrusion Detection Systems: A Comprehensive Survey](https://mlsanity.com/paper/2408.07583/) (2024)
- [GenKubeSec: LLM-Based Kubernetes Misconfiguration Detection, Localization, Reasoning, and Remediation](https://mlsanity.com/paper/2405.19954/) (2024)
- [FAIR Enough: How Can We Develop and Assess a FAIR-Compliant Dataset for Large Language Models' Training?](https://mlsanity.com/paper/2401.11033/) (2024)
- [FinTrust: A Comprehensive Benchmark of Trustworthiness Evaluation in Finance Domain](https://mlsanity.com/paper/2510.15232/) (2025)
- [Adaptive PII Mitigation Framework for Large Language Models](https://mlsanity.com/paper/2501.12465/) (2025)
- [Towards Reliable Detection of LLM-Generated Texts: A Comprehensive Evaluation Framework with CUDRT](https://mlsanity.com/paper/2406.09056/) (2024)
- [Learned, Lagged, LLM-splained: LLM Responses to End User Security Questions](https://mlsanity.com/paper/2411.14571/) (2024)
- [Libra-Leaderboard: Towards Responsible AI through a Balanced Leaderboard of Safety and Capability](https://mlsanity.com/paper/2412.18551/) (2024)
- [Decoding BACnet Packets: A Large Language Model Approach for Packet Interpretation](https://mlsanity.com/paper/2407.15428/) (2024)
- [Mapping Trustworthiness in Large Language Models: A Bibliometric Analysis Bridging Theory to Practice](https://mlsanity.com/paper/2503.04785/) (2025)
- [Assessing LLM Response Quality in the Context of Technology-Facilitated Abuse](https://mlsanity.com/paper/2602.17672/) (2026)
- [Security and Privacy Prompts in the Wild: What Users Ask LLMs and How LLMs Respond](https://mlsanity.com/paper/2606.18062/) (2026)
- [LLaVul: A Multimodal LLM for Interpretable Vulnerability Reasoning about Source Code](https://mlsanity.com/paper/2509.17337/) (2025)
- [Red-Teaming Claude Opus and ChatGPT-based Security Advisors for Trusted Execution Environments](https://mlsanity.com/paper/2602.19450/) (2026)
- [SafePilot: A Framework for Assuring LLM-enabled Cyber-Physical Systems](https://mlsanity.com/paper/2603.21523/) (2026)
- [Semi-Automated Detection of Gaps in LLM Security Knowledge](https://mlsanity.com/paper/2607.18496/) (2026)
- [Large language models in medical time series analysis](https://mlsanity.com/paper/2609.22262/) (2026)

## Abstract (arXiv)

Users seek security & privacy (S&P) advice from online resources, including trusted websites and content-sharing platforms. These resources help users understand S&P technologies and tools and suggest actionable strategies. Large Language Models (LLMs) have recently emerged as trusted information sources. However, their accuracy and correctness have been called into question. Prior research has outlined the shortcomings of LLMs in answering multiple-choice questions and user ability to inadvertently circumvent model restrictions (e.g., to produce toxic content). Yet, the ability of LLMs to provide reliable S&P advice is not well-explored. In this paper, we measure their ability to refute popular S&P misconceptions that the general public holds. We first study recent academic literature to curate a dataset of over a hundred S&P-related misconceptions across six different topics. We then query two popular LLMs (Bard and ChatGPT) and develop a labeling guide to evaluate their responses to these misconceptions. To comprehensively evaluate their responses, we further apply three strategies: query each misconception multiple times, generate and query their paraphrases, and solicit source URLs of the responses. Both models demonstrate, on average, a 21.3% non-negligible error rate, incorrectly supporting popular S&P misconceptions. The error rate increases to 32.6% when we repeatedly query LLMs with the same or paraphrased misconceptions. We also expose that models may partially support a misconception or remain noncommittal, refusing a firm stance on misconceptions. Our exploration of information sources for responses revealed that LLMs are susceptible to providing invalid URLs (21.2% for Bard and 67.7% for ChatGPT) or point to unrelated sources (44.2% returned by Bard and 18.3% by ChatGPT).