# Impact of Guidance and Interaction Strategies for LLM Use on Learner Performance and Perception

arXiv: https://arxiv.org/abs/2310.13712 · HTML: https://mlsanity.com/paper/2310.13712/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 129 references that are in the map.
- First posted on arXiv 13 Oct 2023.

## Builds on (most cited first, 14 of 14)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Large Language Models Are Human-Level Prompt Engineers](https://mlsanity.com/paper/2211.01910/) (2022)
- [Aligning AI With Shared Human Values](https://mlsanity.com/paper/2008.02275/) (2020)
- [How is ChatGPT's behavior changing over time?](https://mlsanity.com/paper/2307.09009/) (2023)
- [AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts](https://mlsanity.com/paper/2110.01691/) (2021)
- [Design Guidelines for Prompt Engineering Text-to-Image Generative Models](https://mlsanity.com/paper/2109.06977/) (2021)
- [Evaluating Human-Language Model Interaction](https://mlsanity.com/paper/2212.09746/) (2022)
- [What Makes a Good Conversation? Challenges in Designing Truly Conversational Agents](https://mlsanity.com/paper/1901.06525/) (2019)
- [Large Language Models and the Reverse Turing Test](https://mlsanity.com/paper/2207.14382/) (2022)
- [Promptly: Using Prompt Problems to Teach Learners How to Effectively Utilize AI Code Generators](https://mlsanity.com/paper/2307.16364/) (2023)
- [Can We Trust AI-Generated Educational Content? Comparative Analysis of Human and AI-Generated Learning Resources](https://mlsanity.com/paper/2306.10509/) (2023)
- [Mapping Perceptions of Humanness in Speech-Based Intelligent Personal Assistant Interaction](https://mlsanity.com/paper/1907.11585/) (2019)
- [ReaderQuizzer: Augmenting Research Papers with Just-In-Time Learning Questions to Facilitate Deeper Understanding](https://mlsanity.com/paper/2308.07988/) (2023)

## Built on by (most cited first, 19 of 19)

- [Should we be going MAD? A Look at Multi-Agent Debate Strategies for LLMs](https://mlsanity.com/paper/2311.17371/) (2023)
- [AI-Augmented Brainwriting: Investigating the use of LLMs in group ideation](https://mlsanity.com/paper/2402.14978/) (2024)
- [Negotiating with LLMS: Prompt Hacks, Skill Gaps, and Reasoning Deficits](https://mlsanity.com/paper/2312.03720/) (2023)
- [SPL: A Socratic Playground for Learning Powered by Large Language Model](https://mlsanity.com/paper/2406.13919/) (2024)
- [StuGPTViz: A Visual Analytics Approach to Understand Student-ChatGPT Interactions](https://mlsanity.com/paper/2407.12423/) (2024)
- [Adaptive Self-Supervised Learning Strategies for Dynamic On-Device LLM Personalization](https://mlsanity.com/paper/2409.16973/) (2024)
- [Exploring Human-LLM Conversations: Mental Models and the Originator of Toxicity](https://mlsanity.com/paper/2407.05977/) (2024)
- [Faculty Perspectives on the Potential of RAG in Computer Science Higher Education](https://mlsanity.com/paper/2408.01462/) (2024)
- [Findings of MEGA: Maths Explanation with LLMs using the Socratic Method for Active Learning](https://mlsanity.com/paper/2507.12079/) (2025)
- [Improving Student-AI Interaction Through Pedagogical Prompting: An Example in Computer Science Education](https://mlsanity.com/paper/2506.19107/) (2025)
- [Transformative Influence of LLM and AI Tools in Student Social Media Engagement: Analyzing Personalization, Communication Efficiency, and Collaborative Learning](https://mlsanity.com/paper/2407.15012/) (2024)
- [A Comparative Study of Student Perspectives on Technical Writing Feedback Quality: Evaluating LLMs, SLMs, and Humans in Computer Science Topics](https://mlsanity.com/paper/2601.11541/) (2025)
- [Exploring Teacher-Chatbot Interaction and Affect in Block-Based Programming](https://mlsanity.com/paper/2603.20211/) (2026)
- [A Scoping Review of Large Language Model-Based Pedagogical Agents](https://mlsanity.com/paper/2604.12253/) (2026)
- [Tracing the Invisible: Understanding Students' Judgment in AI-Supported Design Work](https://mlsanity.com/paper/2505.08939/) (2025)
- [Beyond Autocomplete: Designing CopilotLens Towards Transparent and Explainable AI Coding Agents](https://mlsanity.com/paper/2506.20062/) (2025)
- [Human Thinking under Plural LLM Assistance: Mathematical Problem Solving and Open-Ended Writing](https://mlsanity.com/paper/2604.02677/) (2026)
- [Evaluating Epistemic Guardrails in AI Reading Assistants: A Behavioral Audit of a Minimal Prototype](https://mlsanity.com/paper/2604.27275/) (2026)
- [Feedback Without the Wait: Piloting a Generative AI Practice Platform in a Large Maths Class](https://mlsanity.com/paper/2610.01262/) (2026)

## Abstract (arXiv)

Personalized chatbot-based teaching assistants can be crucial in addressing increasing classroom sizes, especially where direct teacher presence is limited. Large language models (LLMs) offer a promising avenue, with increasing research exploring their educational utility. However, the challenge lies not only in establishing the efficacy of LLMs but also in discerning the nuances of interaction between learners and these models, which impact learners' engagement and results. We conducted a formative study in an undergraduate computer science classroom (N=145) and a controlled experiment on Prolific (N=356) to explore the impact of four pedagogically informed guidance strategies on the learners' performance, confidence and trust in LLMs. Direct LLM answers marginally improved performance, while refining student solutions fostered trust. Structured guidance reduced random queries as well as instances of students copy-pasting assignment questions to the LLM. Our work highlights the role that teachers can play in shaping LLM-supported learning environments.