# The StudyChat Dataset: Analyzing Student Dialogues With ChatGPT in an Artificial Intelligence Course

arXiv: https://arxiv.org/abs/2503.07928 · HTML: https://mlsanity.com/paper/2503.07928/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 63 references that are in the map.
- First posted on arXiv 11 Mar 2025.

## Builds on (most cited first, 18 of 18)

- [MathDial: A Dialogue Tutoring Dataset with Rich Pedagogical Properties Grounded in Math Reasoning Problems](https://mlsanity.com/paper/2305.14536/) (2023)
- [The NCTE Transcripts: A Dataset of Elementary Math Classroom Transcripts](https://mlsanity.com/paper/2211.11772/) (2022)
- [Teach AI How to Code: Using Large Language Models as Teachable Agents for Programming Education](https://mlsanity.com/paper/2309.14534/) (2023)
- [The TalkMoves Dataset: K-12 Mathematics Lesson Transcripts Annotated for Teacher and Student Discursive Moves](https://mlsanity.com/paper/2204.09652/) (2022)
- [Is ChatGPT a Good Teacher Coach? Measuring Zero-Shot Performance For Scoring and Providing Actionable Insights on Classroom Instruction](https://mlsanity.com/paper/2306.03090/) (2023)
- [Jill Watson: A Virtual Teaching Assistant powered by ChatGPT](https://mlsanity.com/paper/2405.11070/) (2024)
- [GPTutor: a ChatGPT-powered programming tool for code explanation](https://mlsanity.com/paper/2305.01863/) (2023)
- [The Teacher-Student Chatroom Corpus](https://mlsanity.com/paper/2011.07109/) (2020)
- [Exploring Automated Distractor Generation for Math Multiple-choice Questions via Large Language Models](https://mlsanity.com/paper/2404.02124/) (2024)
- [Automatically Generating CS Learning Materials with Large Language Models](https://mlsanity.com/paper/2212.05113/) (2022)
- [Improving Reading Comprehension Question Generation with Data Augmentation and Overgenerate-and-rank](https://mlsanity.com/paper/2306.08847/) (2023)
- [Does Informativeness Matter? Active Learning for Educational Dialogue Act Classification](https://mlsanity.com/paper/2304.05578/) (2023)
- [Enhancing Programming Education with ChatGPT: A Case Study on Student Perceptions and Interactions in a Python Course](https://mlsanity.com/paper/2403.15472/) (2024)
- [Detecting Speech Act Types in Developer Question/Answer Conversations During Bug Repair](https://mlsanity.com/paper/1806.05130/) (2018)
- [Robust Educational Dialogue Act Classifiers with Low-Resource and Imbalanced Datasets](https://mlsanity.com/paper/2304.07499/) (2023)
- [Kattis vs. ChatGPT: Assessment and Evaluation of Programming Tasks in the Age of Artificial Intelligence](https://mlsanity.com/paper/2312.01109/) (2023)
- [Exploring Automated Keyword Mnemonics Generation with Large Language Models via Overgenerate-and-Rank](https://mlsanity.com/paper/2409.13952/) (2024)
- [Assessing the Promise and Pitfalls of ChatGPT for Automated Code Generation](https://mlsanity.com/paper/2311.02640/) (2023)

## Built on by (most cited first, 9 of 9)

- [Do Students Rely on AI? Analysis of Student-ChatGPT Conversations from a Field Study](https://mlsanity.com/paper/2508.20244/) (2025)
- ["I Didn't Make the Micro Decisions": Measuring, Inducing, and Exposing Goal-Level AI Contributions in Collaboration](https://mlsanity.com/paper/2605.21363/) (2026)
- [LLM Pedagogical Behavior in AI Tutoring Interactions](https://mlsanity.com/paper/2608.22993/) (2026)
- [DIAL-SUMMER: A Structured Evaluation Framework of Hierarchical Errors in Dialogue Summaries](https://mlsanity.com/paper/2602.08149/) (2026)
- [Toward Measuring AI's Effects on Skill Formation: The Stock-Formation Gap](https://mlsanity.com/paper/2605.16283/) (2026)
- [PIVOT: Preference-based Intervention Vectors for Pedagogical Tutor Steering](https://mlsanity.com/paper/2608.07509/) (2026)
- [Principal Trait Analysis: Towards Deriving "Skills" in Human-AI Collaboration](https://mlsanity.com/paper/2608.11460/) (2026)
- [UTILMEM: Benchmarking Evidence Utilization in Long-Term Conversational Memory](https://mlsanity.com/paper/2608.30508/) (2026)
- [Dataset Signatures in Human-LLM Interactions and User Modeling](https://mlsanity.com/paper/2610.05534/) (2026)

## Abstract (arXiv)

The widespread availability of large language models (LLMs), such as ChatGPT, has significantly impacted education, raising both opportunities and challenges. Students can frequently interact with LLM-powered, interactive learning tools, but their usage patterns need to be observed and understood. We introduce StudyChat, a publicly available dataset capturing real-world student interactions with an LLM-powered tutoring chatbot in a semester-long, university-level artificial intelligence (AI) course. We deploy a web application that replicates ChatGPT's core functionalities, and use it to log student interactions with the LLM while working on programming assignments. We collect 16,851 interactions, which we annotate using a dialogue act labeling schema inspired by observed interaction patterns and prior research. We analyze these interactions, highlight usage trends, and analyze how specific student behavior correlates with their course outcome. We find that students who prompt LLMs for conceptual understanding and coding help tend to perform better on assignments and exams. Moreover, students who use LLMs to write reports and circumvent assignment learning objectives have lower outcomes on exams than others. StudyChat serves as a shared resource to facilitate further research on the evolving role of LLMs in education.