# Chat-Based Support Alone May Not Be Enough: Comparing Conversational and Embedded LLM Feedback for Mathematical Proof Learning

arXiv: https://arxiv.org/abs/2602.18807 · HTML: https://mlsanity.com/paper/2602.18807/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 38 references that are in the map.
- First posted on arXiv 21 Feb 2026.

## Builds on (most cited first, 14 of 14)

- [Mathematical Capabilities of ChatGPT](https://mlsanity.com/paper/2301.13867/) (2023)
- [Explanations Can Reduce Overreliance on AI Systems During Decision-Making](https://mlsanity.com/paper/2212.06823/) (2022)
- [Appropriate Reliance on AI Advice: Conceptualization and the Effect of Explanations](https://mlsanity.com/paper/2302.02187/) (2023)
- [Bringing Generative AI to Adaptive Learning in Education](https://mlsanity.com/paper/2402.14601/) (2024)
- [GPTutor: a ChatGPT-powered programming tool for code explanation](https://mlsanity.com/paper/2305.01863/) (2023)
- [Improving Assessment of Tutoring Practices using Retrieval-Augmented Generation](https://mlsanity.com/paper/2402.14594/) (2024)
- [How Can I Improve? Using GPT to Highlight the Desired and Undesired Parts of Open-ended Responses](https://mlsanity.com/paper/2405.00291/) (2024)
- [GPTutor: Great Personalized Tutor with Large Language Models for Personalized Learning Content Generation](https://mlsanity.com/paper/2407.09484/) (2024)
- [A Systematic Review on Prompt Engineering in Large Language Models for K-12 STEM Education](https://mlsanity.com/paper/2410.11123/) (2024)
- [SlideItRight: Using AI to Find Relevant Slides and Provide Feedback for Open-Ended Questions](https://mlsanity.com/paper/2505.04584/) (2025)
- [VTutor for High-Impact Tutoring at Scale: Managing Engagement and Real-Time Multi-Screen Monitoring with P2P Connections](https://mlsanity.com/paper/2505.07736/) (2025)
- [VTutor: An Animated Pedagogical Agent SDK that Provide Real Time Multi-Model Feedback](https://mlsanity.com/paper/2505.06676/) (2025)
- [AI Knows Best? The Paradox of Expertise, AI-Reliance, and Performance in Educational Tutoring Decision-Making Tasks](https://mlsanity.com/paper/2509.16772/) (2025)
- [Generative AI alone may not be enough: Evaluating AI Support for Learning Mathematical Proof](https://mlsanity.com/paper/2509.16778/) (2025)

## Abstract (arXiv)

We evaluate GPTutor, an LLM-powered tutoring system for an undergraduate discrete mathematics course. It integrates two LLM-supported tools: a structured proof-review tool that provides embedded feedback on students' written proof attempts, and a chatbot for math questions. In a staggered-access study with 148 students, earlier access was associated with higher homework performance during the interval when only the experimental group could use the system, while we did not observe this performance increase transfer to exam scores. Usage logs show that students with lower self-efficacy and prior exam performance used both components more frequently. Session-level behavioral labels, produced by human coding and scaled using an automated classifier, characterize how students engaged with the chatbot (e.g., answer-seeking or help-seeking). In models controlling for prior performance and self-efficacy, higher chatbot usage and answer-seeking behavior were negatively associated with subsequent midterm performance, whereas proof-review usage showed no detectable independent association. Together, the findings suggest that chatbot-based support alone may not reliably support transfer to independent assessment of math proof-learning outcomes, whereas work-anchored, structured feedback appears less associated with reduced learning.