# Why and When LLM-Based Assistants Can Go Wrong: Investigating the Effectiveness of Prompt-Based Interactions for Software Help-Seeking arXiv: https://arxiv.org/abs/2402.08030 ยท HTML: https://mlsanity.com/paper/2402.08030/ ## Key facts - Cited by 26 papers in the map (data through 14 Oct 2026). - Builds on 13 of its 53 references that are in the map. - First posted on arXiv 12 Feb 2024. ## Builds on (most cited first, 13 of 13) - [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020) - [Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning](https://mlsanity.com/paper/2205.05638/) (2022) - [A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity](https://mlsanity.com/paper/2302.04023/) (2023) - [Instruction Tuning for Large Language Models: A Survey](https://mlsanity.com/paper/2308.10792/) (2023) - [Grounded Copilot: How Programmers Interact with Code-Generating Models](https://mlsanity.com/paper/2206.15000/) (2022) - [A Categorical Archive of ChatGPT Failures](https://mlsanity.com/paper/2302.03494/) (2023) - [AI Transparency in the Age of LLMs: A Human-Centered Research Roadmap](https://mlsanity.com/paper/2306.01941/) (2023) - [Explanations Can Reduce Overreliance on AI Systems During Decision-Making](https://mlsanity.com/paper/2212.06823/) (2022) - [Is ChatGPT the Ultimate Programming Assistant -- How far is it?](https://mlsanity.com/paper/2304.11938/) (2023) - [In Search of Verifiability: Explanations Rarely Enable Complementary Performance in AI-Advised Decision Making](https://mlsanity.com/paper/2305.07722/) (2023) - [Investigating Explainability of Generative AI for Code through Scenario-based Design](https://mlsanity.com/paper/2202.04903/) (2022) - [In-IDE Code Generation from Natural Language: Promise and Challenges](https://mlsanity.com/paper/2101.11149/) (2021) - [What is it like to program with artificial intelligence?](https://mlsanity.com/paper/2208.06213/) (2022) ## Built on by (most cited first, 26 of 26) - [Understanding the Dark Side of LLMs' Intrinsic Self-Correction](https://mlsanity.com/paper/2412.14959/) (2024) - [VeriPlan: Integrating Formal Verification and LLMs into End-User Planning](https://mlsanity.com/paper/2502.17898/) (2025) - [A Study of LLMs' Preferences for Libraries and Programming Languages](https://mlsanity.com/paper/2503.17181/) (2025) - [LLM Whisperer: An Inconspicuous Attack to Bias LLM Responses](https://mlsanity.com/paper/2406.04755/) (2024) - [LLM Agent Meets Agentic AI: Can LLM Agents Simulate Customers to Evaluate Agentic-AI-based Shopping Assistants?](https://mlsanity.com/paper/2509.21501/) (2025) - [Conversational User-AI Intervention: A Study on Prompt Rewriting for Improved LLM Response Generation](https://mlsanity.com/paper/2503.16789/) (2025) - [DICE-BENCH: Evaluating the Tool-Use Capabilities of Large Language Models in Multi-Round, Multi-Party Dialogues](https://mlsanity.com/paper/2506.22853/) (2025) - [Do It For Me vs. Do It With Me: Investigating User Perceptions of Different Paradigms of Automation in Copilots for Feature-Rich Software](https://mlsanity.com/paper/2504.15549/) (2025) - [Conversational Agents as Catalysts for Critical Thinking: Challenging Design Fixation in Group Design](https://mlsanity.com/paper/2406.11125/) (2024) - [Navigating the Unknown: A Chat-Based Collaborative Interface for Personalized Exploratory Tasks](https://mlsanity.com/paper/2410.24032/) (2024) - [Why is prompting hard? Understanding prompts on binary sequence predictors](https://mlsanity.com/paper/2502.10760/) (2025) - [GUIDE: A Benchmark for Understanding and Assisting Users in Open-Ended GUI Tasks](https://mlsanity.com/paper/2603.25864/) (2026) - [Navigating User Experience of ChatGPT-based Conversational Recommender Systems: The Effects of Prompt Guidance and Recommendation Domain](https://mlsanity.com/paper/2405.13560/) (2024) - [Unveiling Large Language Models Generated Texts: A Multi-Level Fine-Grained Detection Framework](https://mlsanity.com/paper/2410.14231/) (2024) - [A Crowdsourced Study of ChatBot Influence in Value-Driven Decision Making Scenarios](https://mlsanity.com/paper/2511.15857/) (2025) - [A Risk Taxonomy and Reflection Tool for Large Language Model Adoption in Public Health](https://mlsanity.com/paper/2411.02594/) (2024) - [ChatGPT and U(X): A Rapid Review on Measuring the User Experience](https://mlsanity.com/paper/2503.15808/) (2025) - [Evaluating the Effectiveness of Large Language Models in Solving Simple Programming Tasks: A User-Centered Study](https://mlsanity.com/paper/2507.04043/) (2025) - [Growth First, Care Second? Tracing the Landscape of LLM Value Preferences in Everyday Dilemmas](https://mlsanity.com/paper/2602.04456/) (2026) - [Helping the Helper: LLM-Assisted Problem Articulation for Older Adults Seeking Technology Support](https://mlsanity.com/paper/2601.10018/) (2026) - [TreeWriter: AI-Assisted Hierarchical Planning and Writing for Long-Form Documents](https://mlsanity.com/paper/2601.12740/) (2026) - [PromptHelper: A Prompt Recommender System for Encouraging Creativity in AI Chatbot Interactions](https://mlsanity.com/paper/2601.15575/) (2026) - ["I Just Need GPT to Refine My Prompts": Rethinking Onboarding and Help-Seeking with Generative 3D Modeling Tools](https://mlsanity.com/paper/2603.29118/) (2026) - [Beyond Chat and Clicks: GUI Agents for In-Situ Assistance via Live Interface Transformation](https://mlsanity.com/paper/2604.14668/) (2026) - [Evaluating the Effects of Prompt Perturbation on Bias and Hallucination in Large Language Models](https://mlsanity.com/paper/2609.35804/) (2026) - [DataWeave: Deploying Human-LLM Analytics for Exploratory Structured Data Analysis](https://mlsanity.com/paper/2610.02679/) (2026) ## Abstract (arXiv) Large Language Model (LLM) assistants, such as ChatGPT, have emerged as potential alternatives to search methods for helping users navigate complex, feature-rich software. LLMs use vast training data from domain-specific texts, software manuals, and code repositories to mimic human-like interactions, offering tailored assistance, including step-by-step instructions. In this work, we investigated LLM-generated software guidance through a within-subject experiment with 16 participants and follow-up interviews. We compared a baseline LLM assistant with an LLM optimized for particular software contexts, SoftAIBot, which also offered guidelines for constructing appropriate prompts. We assessed task completion, perceived accuracy, relevance, and trust. Surprisingly, although SoftAIBot outperformed the baseline LLM, our results revealed no significant difference in LLM usage and user perceptions with or without prompt guidelines and the integration of domain context. Most users struggled to understand how the prompt's text related to the LLM's responses and often followed the LLM's suggestions verbatim, even if they were incorrect. This resulted in difficulties when using the LLM's advice for software tasks, leading to low task completion rates. Our detailed analysis also revealed that users remained unaware of inaccuracies in the LLM's responses, indicating a gap between their lack of software expertise and their ability to evaluate the LLM's assistance. With the growing push for designing domain-specific LLM assistants, we emphasize the importance of incorporating explainable, context-aware cues into LLMs to help users understand prompt-based interactions, identify biases, and maximize the utility of LLM assistants.