Learned, Lagged, LLM-splained: LLM Responses to End User Security Questions
Not on the map. Why?
Abstract (arXiv)
Answering end user security questions is challenging. While large language models (LLMs) like GPT, LLAMA, and Gemini are far from error-free, they have shown promise in answering a variety of questions outside of security. We studied LLM performance in the area of end user security by qualitatively evaluating 3 popular LLMs on 900 systematically collected end user security questions. While LLMs demonstrate broad generalist ``knowledge'' of end user security information, there are patterns of errors and limitations across LLMs consisting of stale and inaccurate answers, and indirect or unresponsive communication styles, all of which impacts the quality of information received. Based on these patterns, we suggest directions for model improvement and recommend user strategies for interacting with LLMs when seeking assistance with security.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 22 of its 99 references that are in the map. First posted on arXiv 21 Nov 2024.
Builds on
22 of 99 references.
References · 22
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2023GPT-4 Technical Reportcited by 18,396
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
- 2020Measuring Massive Multitask Language Understandingcited by 7,963
- 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
- 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
- 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
- 2023A Survey on Evaluation of Large Language Modelscited by 1,242
- 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,207
- 2023Towards Understanding Sycophancy in Language Modelscited by 1,040
- 2020Learning under Concept Drift: A Reviewcited by 477
- 2020Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performancecited by 399
10 more
- 2019Explain Yourself! Leveraging Language Models for Commonsense Reasoningcited by 367
- 2023SafetyBench: Evaluating the Safety of Large Language Modelscited by 199
- 2020Unsupervised Commonsense Question Answering with Self-Talkcited by 161
- 2023Purple Llama CyberSecEval: A Secure Coding Benchmark for Language Modelscited by 138
- 2023"It's a Fair Game", or Is It? Examining How Users Navigate Disclosure Risks and Benefits When Using LLM-Based Conversational Agentscited by 60
- 2023Can Large Language Models Provide Security & Privacy Advice? Measuring the Ability of LLMs to Refute Misconceptionscited by 20
- 2023AI Reliance and Decision Quality: Fundamentals, Interdependence, and the Effects of Interventionscited by 9
- 2022A World Full of Privacy and Security (Mis)conceptions? Findings of a Representative Survey in 12 Countriescited by 3
- 2020Costs and benefits of authentication advicecited by 1
- 2022Perspectives of Non-Expert Users on Cyber Security and Privacy: An Analysis of Online Discussions on Twittercited by 1
Built on by
Citing papers per year: 2026: 2.
2 papers.