mlsanityData to 7 Oct 2026

Learning gain differences between ChatGPT and human tutor generated algebra hints

arXiv 2302.06871 ↗First posted 14 Feb 2023cs.CY cs.CL cs.HCcited by 42

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs), such as ChatGPT, are quickly advancing AI to the frontiers of practical consumer use and leading industries to re-evaluate how they allocate resources for content production. Authoring of open educational resources and hint content within adaptive tutoring systems is labor intensive. Should LLMs like ChatGPT produce educational content on par with human-authored content, the implications would be significant for further scaling of computer tutoring system approaches. In this paper, we conduct the first learning gain evaluation of ChatGPT by comparing the efficacy of its hints with hints authored by human tutors with 77 participants across two algebra topic areas, Elementary Algebra and Intermediate Algebra. We find that 70% of hints produced by ChatGPT passed our manual quality checks and that both human and ChatGPT conditions produced positive learning gains. However, gains were only statistically significant for human tutor created hints. Learning gains from human-created hints were substantially and statistically significantly higher than ChatGPT hints in both topic areas, though ChatGPT participants in the Intermediate Algebra experiment were near ceiling and not even with the control at pre-test. We discuss the limitations of our study and suggest several future directions for the field. Problem and hint content used in the experiment is provided for replicability.

Key facts. Cited by 42 papers in the map (data through 7 Oct 2026). Builds on 13 of its 31 references that are in the map. First posted on arXiv 14 Feb 2023.

Builds on

13 of 31 references.

References · 13

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  7. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  8. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  9. 2017Deep reinforcement learning from human preferencescited by 3,846
  10. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 895
  11. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 691
  12. 2023ChatGPT is not all you need. A State of the Art Review of large Generative AI modelscited by 85
1 more
  1. 2022Automatically Generating CS Learning Materials with Large Language Modelscited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 19; 2024: 14; 2025: 5; 2026: 4.

42 papers.

Sort:

By area:

  1. 2023Evaluating Large Language Models: A Comprehensive Surveycited by 200
  2. 2024Large Language Models for Education: A Survey and Outlookcited by 185
  3. 2023Summary of ChatGPT-Related Research and Perspective Towards the Future of Large Language Modelscited by 152
  4. 2023Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Surveycited by 134
  5. 2023Should we be going MAD? A Look at Multi-Agent Debate Strategies for LLMscited by 103
  6. 2023Is ChatGPT a Good Teacher Coach? Measuring Zero-Shot Performance For Scoring and Providing Actionable Insights on Classroom Instructioncited by 37
  7. 2023Large Language Models (GPT) for automating feedback on programming assignmentscited by 32
  8. 2024Leveraging LLM-Respondents for Item Evaluation: a Psychometric Analysiscited by 31
  9. 2024Ruffle&Riley: Insights from Designing and Evaluating a Large Language Model-Based Conversational Tutoring Systemcited by 25
  10. 2023Is AI the better programming partner? Human-Human Pair Programming vs. Human-AI pAIr Programmingcited by 20
  11. 2024Towards the Pedagogical Steering of Large Language Models for Tutoring: A Case Study with Modeling Productive Failurecited by 20
  12. 2023Evaluating ChatGPT's Decimal Skills and Feedback Generation in a Digital Learning Gamecited by 19
30 more
  1. 2023Let's have a chat! A Conversation with ChatGPT: Technology, Applications, and Limitationscited by 16
  2. 2023ChatGPT is a Remarkable Tool -- For Expertscited by 16
  3. 2025EduBench: A Comprehensive Benchmarking Dataset for Evaluating Large Language Models in Diverse Educational Scenarioscited by 16
  4. 2024The GPT Surprise: Offering Large Language Model Chat in a Massive Coding Class Reduced Engagement but Increased Adopters Exam Performancescited by 14
  5. 2023Ruffle&Riley: Towards the Automated Induction of Conversational Tutoring Systemscited by 12
  6. 2023Unlocking the Potential of ChatGPT: A Comprehensive Exploration of its Applications, Advantages, Limitations, and Future Directions in Natural Language Processingcited by 11
  7. 2024Evaluating and Optimizing Educational Content with Large Language Model Judgmentscited by 11
  8. 2024AI-Tutoring in Software Engineering Educationcited by 11
  9. 2025A Novel Approach to Scalable and Automatic Topic-Controlled Question Generation in Educationcited by 11
  10. 2024PromptHive: Bringing Subject Matter Experts Back to the Forefront with Collaborative Prompt Engineering for Educational Content Creationcited by 9
  11. 2023Exploring the Use of ChatGPT as a Tool for Learning and Assessment in Undergraduate Computer Science Curriculum: Opportunities and Challengescited by 7
  12. 2023Automated Evaluation of Classroom Instructional Support with LLMs and BoWs: Connecting Global Predictions to Specific Feedbackcited by 7
  13. 2023Early ChatGPT User Portrait through the Lens of Datacited by 6
  14. 2023Students' Perceptions and Preferences of Generative Artificial Intelligence Feedback for Programmingcited by 6
  15. 2024AI Meets the Classroom: When Do Large Language Models Harm Learning?cited by 5
  16. 2026Evaluating Answer Leakage Robustness of LLM Tutors against Adversarial Student Attackscited by 5
  17. 2023Diversity Measures: Domain-Independent Proxies for Failure in Language Model Queriescited by 4
  18. 2024When Geoscience Meets Generative AI and Large Language Models: Foundations, Trends, and Future Challengescited by 4
  19. 2024Large Language Models Meet User Interfaces: The Case of Provisioning Feedbackcited by 4
  20. 2024Towards Generalizable Agents in Text-Based Educational Environments: A Study of Integrating RL with LLMscited by 4
  21. 2025CoDAE: Adapting Large Language Models for Education via Chain-of-Thought Data Augmentationcited by 4
  22. 2023Scaling Evidence-based Instructional Design Expertise through Large Language Modelscited by 3
  23. 2023Unmasking the giant: A comprehensive evaluation of ChatGPT's proficiency in coding algorithms and data structurescited by 3
  24. 2025LeanTutor: Towards a Verified AI Mathematical Proof Tutorcited by 3
  25. 2026Knowledge without Wisdom: Measuring Misalignment between LLMs and Intended Impactcited by 2
  26. 2024SeSaMe: A Framework to Simulate Self-Reported Ground Truth for Mental Health Sensing Studiescited by 1
  27. 2025Small but Significant: On the Promise of Small Language Models for Accessible AIEDcited by 1
  28. 2026AI-Mediated Feedback Improves Student Revisions: A Randomized Trial with FeedbackWriter in a Large Undergraduate Coursecited by 1
  29. 2024Investigation of the effectiveness of applying ChatGPT in Dialogic Teaching Using Electroencephalographycited by 0
  30. 2026LeanTutor: Towards a Verified AI Mathematical Proof Tutorcited by 0