mlsanityData to 13 Oct 2026

Comparing the Utility, Preference, and Performance of Course Material Search Functionality and Retrieval-Augmented Generation Large Language Model (RAG-LLM) AI Chatbots in Information-Seeking Tasks

arXiv 2410.13326 ↗First posted 17 Oct 2024cs.CY cs.IRcited by 0

On the map: Retrieval-augmented generation → Agentic RAG, agents and memory

Abstract (arXiv)

Providing sufficient support for students requires substantial resources, especially considering the growing enrollment numbers. Students need help in a variety of tasks, ranging from information-seeking to requiring support with course assignments. To explore the utility of recent large language models (LLMs) as a support mechanism, we developed an LLM-powered AI chatbot that augments the answers that are produced with information from the course materials. To study the effect of the LLM-powered AI chatbot, we conducted a lab-based user study (N=14), in which the participants worked on tasks from a web software development course. The participants were divided into two groups, where one of the groups first had access to the chatbot and then to a more traditional search functionality, while another group started with the search functionality and was then given the chatbot. We assessed the participants' performance and perceptions towards the chatbot and the search functionality and explored their preferences towards the support functionalities. Our findings highlight that both support mechanisms are seen as useful and that support mechanisms work well for specific tasks, while less so for other tasks. We also observe that students tended to prefer the second support mechanism more, where students who were first given the chatbot tended to prefer the search functionality and vice versa.

Key facts. No citing papers in the map yet (data through 13 Oct 2026). Placed in Agentic RAG, agents and memory, one of 15 directions in Retrieval-augmented generation. Builds on 15 of its 26 references that are in the map. First posted on arXiv 17 Oct 2024.

Builds on

15 of 26 references.

References · 15

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,221
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,205
  4. 2023The Robots are Here: Navigating the Generative AI Revolution in Computing Educationcited by 86
  5. 2022Experiences from Using Code Explanations Generated by Large Language Models in a Web Software Development E-Bookcited by 84
  6. 2022Conversing with Copilot: Exploring Prompt Engineering for Solving CS1 Problems Using Natural Languagecited by 82
  7. 2023CodeHelp: Using Large Language Models with Guardrails for Scalable Support in Programming Classescited by 72
  8. 2024A Complete Survey on LLM-based AI Chatbotscited by 71
  9. 2022Using Large Language Models to Enhance Programming Error Messagescited by 67
  10. 2023Computing Education in the Era of Generative AIcited by 52
  11. 2023Exploring the Responses of Large Language Models to Beginner Programmers' Help Requestscited by 46
  12. 2023AI-TA: Towards an Intelligent Question-Answer Teaching Assistant using Open-Source LLMscited by 22
3 more
  1. 2023"AI enhances our performance, I have no doubt this one will do the same": The Placebo effect is robust to negative descriptions of AIcited by 11
  2. 2024Explaining Code with a Purpose: An Integrated Approach for Developing Code Comprehension and Prompting Skillscited by 11
  3. 2024Experiences from Integrating Large Language Model Chatbots into the Classroomcited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Large Language Models in Teaching and Learning: Reflections on Implementing an AI Chatbot in Higher Education ↗
  2. 2025From Textbook to Talkbot: A Case Study of a Greek-Language RAG-Based Chatbot in Higher Education
  3. 2025A Knowledge Graph and a Tripartite Evaluation Framework Make Retrieval-Augmented Generation Scalable and Transparent
  4. 2025From Slides to Chatbots: Enhancing Large Language Models with University Course Materials
  5. 2024Evaluating the Impact of Advanced LLM Techniques on AI-Lecture Tutors for a Robotics Course
  6. 2025Towards Efficient Educational Chatbots: Benchmarking RAG Frameworks
  7. 2024Evaluating the Retrieval Component in LLM-Based Question Answering Systems
  8. 2024Reinforcement Learning for Optimizing RAG for Domain Chatbots