# Understanding Learner-LLM Chatbot Interactions and the Impact of Prompting Guidelines

arXiv: https://arxiv.org/abs/2504.07840 · HTML: https://mlsanity.com/paper/2504.07840/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 32 references that are in the map.
- First posted on arXiv 10 Apr 2025.

## Builds on (most cited first, 13 of 13)

- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT](https://mlsanity.com/paper/2302.11382/) (2023)
- [How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources](https://mlsanity.com/paper/2306.04751/) (2023)
- [Exploring the Limits of ChatGPT for Query or Aspect-based Text Summarization](https://mlsanity.com/paper/2302.08081/) (2023)
- [Generative AI](https://mlsanity.com/paper/2309.07930/) (2023)
- [MIDAS: A Dialog Act Annotation Scheme for Open Domain Human Machine Spoken Conversations](https://mlsanity.com/paper/1908.10023/) (2019)
- [Developing Effective Educational Chatbots with ChatGPT prompts: Insights from Preliminary Tests in a Case Study on Social Media Literacy (with appendix)](https://mlsanity.com/paper/2306.10645/) (2023)
- [Learning to Prompt in the Classroom to Understand AI Limits: A pilot study](https://mlsanity.com/paper/2307.01540/) (2023)
- [Grimm in Wonderland: Prompt Engineering with Midjourney to Illustrate Fairytales](https://mlsanity.com/paper/2302.08961/) (2023)
- [Unimib Assistant: designing a student-friendly RAG-based chatbot for all their needs](https://mlsanity.com/paper/2411.19554/) (2024)
- [Use Me Wisely: AI-Driven Assessment for LLM Prompting Skills Development](https://mlsanity.com/paper/2503.02532/) (2025)

## Built on by (most cited first, 6 of 6)

- [Animating Language Practice: Engagement with Stylized Conversational Agents in Japanese Learning](https://mlsanity.com/paper/2507.06483/) (2025)
- [What Students Ask, How a Generative AI Assistant Responds: Exploring Higher Education Students' Dialogues on Learning Analytics Feedback](https://mlsanity.com/paper/2601.04919/) (2026)
- [The Hidden Cost of Contextual Sycophancy: an AI Literacy Intervention in Human-AI Collaboration](https://mlsanity.com/paper/2605.18372/) (2026)
- ["How Do I ...?": Procedural Questions Predominate Student-LLM Chatbot Conversations](https://mlsanity.com/paper/2602.18372/) (2026)
- [ASSEMCAD: Production-Ready CAD Assembly Generation from Natural Language](https://mlsanity.com/paper/2607.05123/) (2026)
- [Designing Human-mediated AI Guidance: Ready Together for Personalized Family Emergency Preparedness](https://mlsanity.com/paper/2608.19950/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have transformed human-computer interaction by enabling natural language-based communication with AI-powered chatbots. These models are designed to be intuitive and user-friendly, allowing users to articulate requests with minimal effort. However, despite their accessibility, studies reveal that users often struggle with effective prompting, resulting in inefficient responses. Existing research has highlighted both the limitations of LLMs in interpreting vague or poorly structured prompts and the difficulties users face in crafting precise queries. This study investigates learner-AI interactions through an educational experiment in which participants receive structured guidance on effective prompting. We introduce and compare three types of prompting guidelines: a task-specific framework developed through a structured methodology and two baseline approaches. To assess user behavior and prompting efficacy, we analyze a dataset of 642 interactions from 107 users. Using Von NeuMidas, an extended pragmatic annotation schema for LLM interaction analysis, we categorize common prompting errors and identify recurring behavioral patterns. We then evaluate the impact of different guidelines by examining changes in user behavior, adherence to prompting strategies, and the overall quality of AI-generated responses. Our findings provide a deeper understanding of how users engage with LLMs and the role of structured prompting guidance in enhancing AI-assisted communication. By comparing different instructional frameworks, we offer insights into more effective approaches for improving user competency in AI interactions, with implications for AI literacy, chatbot usability, and the design of more responsive AI systems.