mlsanityData to 14 Oct 2026

Investigating Student Interaction Patterns with Large Language Model-Powered Course Assistants in Computer Science Courses

arXiv 2509.08862 ↗First posted 10 Sep 2025cs.CY cs.AI cs.HCcited by 0

Not on the map. Why?

Abstract (arXiv)

Providing students with flexible and timely academic support is a challenge at most colleges and universities, leaving many students without help outside scheduled hours. Large language models (LLMs) are promising for bridging this gap, but interactions between students and LLMs are rarely overseen by educators. We developed and studied an LLM-powered course assistant deployed across multiple computer science courses to characterize real-world use and understand pedagogical implications. By Spring 2024, our system had been deployed to approximately 2,000 students across six courses at three institutions. Analysis of the interaction data shows that usage remains strong in the evenings and nights and is higher in introductory courses, indicating that our system helps address temporal support gaps and novice learner needs. We sampled 200 conversations per course for manual annotation: most sampled responses were judged correct and helpful, with a small share unhelpful or erroneous; few responses included dedicated examples. We also examined an inquiry-based learning strategy: only around 11% of sampled conversations contained LLM-generated follow-up questions, which were often ignored by students in advanced courses. A Bloom's taxonomy analysis reveals that current LLM capabilities are limited in generating higher-order cognitive questions. These patterns suggest opportunities for pedagogically oriented LLM-based educational systems and greater educator involvement in configuring prompts, content, and policies.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 34 of its 65 references that are in the map. First posted on arXiv 10 Sep 2025.

Builds on

34 of 65 references.

References · 34

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  3. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  4. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
  5. 2023A Survey on Evaluation of Large Language Modelscited by 1,242
  6. 2024GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Modelscited by 550
  7. 2023Ragas: Automated Evaluation of Retrieval Augmented Generationcited by 418
  8. 2024Hallucination is Inevitable: An Innate Limitation of Large Language Modelscited by 332
  9. 2024Large Language Models for Education: A Survey and Outlookcited by 188
  10. 2022A Survey on Retrieval-Augmented Text Generationcited by 158
  11. 2023Evaluating ChatGPT's Information Extraction Capabilities: An Assessment of Performance, Explainability, Calibration, and Faithfulnesscited by 130
  12. 2022Improving the Domain Adaptation of Retrieval Augmented Generation (RAG) Models for Open Domain Question Answeringcited by 127
22 more
  1. 2022Automatic Generation of Programming Exercises and Code Explanations using Large Language Modelscited by 126
  2. 2023Table Meets LLM: Can Large Language Models Understand Structured Table Data? A Benchmark and Empirical Studycited by 116
  3. 2023When Large Language Models Meet Personalization: Perspectives of Challenges and Opportunitiescited by 90
  4. 2024CodeAid: Evaluating a Classroom Deployment of an LLM-based Programming Assistant that Balances Student and Educator Needscited by 84
  5. 2023EduChat: A Large-Scale Language Model-based Chatbot System for Intelligent Educationcited by 80
  6. 2024A Complete Survey on LLM-based AI Chatbotscited by 71
  7. 2024LogicBench: Towards Systematic Evaluation of Logical Reasoning Ability of Large Language Modelscited by 68
  8. 2023How Novices Use LLM-Based Code Generators to Solve CS1 Coding Tasks in a Self-Paced Learning Environmentcited by 56
  9. 2023Leveraging Large Language Models to Power Chatbots for Collecting User Self-Reported Datacited by 48
  10. 2023Students' Voices on Generative AI: Perceptions, Benefits, and Challenges in Higher Educationcited by 41
  11. 2023Evaluating Large Language Models on Graphs: Performance Insights and Comparative Analysiscited by 35
  12. 2024LLMs are Biased Teachers: Evaluating LLM Bias in Personalized Educationcited by 28
  13. 2025"Don't Forget the Teachers": Towards an Educator-Centered Understanding of Harms from Large Language Models in Educationcited by 14
  14. 2024Investigating Interaction Modes and User Agency in Human-LLM Collaboration for Domain-Specific Data Analysiscited by 12
  15. 2024Student-AI Interaction: A Case Study of CS1 studentscited by 12
  16. 2024Mindful-RAG: A Study of Points of Failure in Retrieval Augmented Generationcited by 12
  17. 2023Assessing the efficacy of large language models in generating accurate teacher responsescited by 6
  18. 2024Student Perspectives on Using a Large Language Model (LLM) for an Assignment on Professional Ethicscited by 6
  19. 2023Is a Seat at the Table Enough? Engaging Teachers and Students in Dataset Specification for ML in Educationcited by 5
  20. 2025A Framework for Auditing Chatbots for Dialect-Based Quality-of-Service Harmscited by 5
  21. 2021Characterizing Student Engagement Moods for Dropout Prediction in Question Pool Websitescited by 4
  22. 2024How Do Students Interact with an LLM-powered Virtual Teaching Assistant in Different Educational Settings?cited by 4

Built on by

Not cited yet.