# LLM-Generated Feedback Supports Learning If Learners Choose to Use It

arXiv: https://arxiv.org/abs/2506.17006 · HTML: https://mlsanity.com/paper/2506.17006/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 33 references that are in the map.
- First posted on arXiv 20 Jun 2025.

## Builds on (most cited first, 8 of 8)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Beware of Metacognitive Laziness: Effects of Generative Artificial Intelligence on Learning Motivation, Processes, and Performance](https://mlsanity.com/paper/2412.09315/) (2024)
- [Enhancing LLM-Based Feedback: Insights from Intelligent Tutoring Systems and the Learning Sciences](https://mlsanity.com/paper/2405.04645/) (2024)
- [Can We Trust AI-Generated Educational Content? Comparative Analysis of Human and AI-Generated Learning Resources](https://mlsanity.com/paper/2306.10509/) (2023)
- [How Can I Get It Right? Using GPT to Rephrase Incorrect Trainee Responses](https://mlsanity.com/paper/2405.00970/) (2024)
- [How Can I Improve? Using GPT to Highlight the Desired and Undesired Parts of Open-ended Responses](https://mlsanity.com/paper/2405.00291/) (2024)
- [Do Tutors Learn from Equity Training and Can Generative AI Assess It?](https://mlsanity.com/paper/2412.11255/) (2024)
- [GEEPERs: Principal Stratification using Principal Scores and Stacked Estimating Equations](https://mlsanity.com/paper/2212.10406/) (2022)

## Built on by (most cited first, 4 of 4)

- [Evidence-Decision-Feedback: Theory-Driven Adaptive Scaffolding for LLM Agents](https://mlsanity.com/paper/2602.01415/) (2026)
- [AI-Mediated Feedback Improves Student Revisions: A Randomized Trial with FeedbackWriter in a Large Undergraduate Course](https://mlsanity.com/paper/2602.16820/) (2026)
- [REFINE: Real-world Exploration of Interactive Feedback and Student Behaviour](https://mlsanity.com/paper/2603.29142/) (2026)
- [AI-Driven Assessment of Human Tutors: Linking Training Performance to Real-Life Practice](https://mlsanity.com/paper/2606.18617/) (2026)

## Abstract (arXiv)

Large language models (LLMs) are increasingly used to generate feedback, yet their impact on learning remains underexplored, especially compared to existing feedback methods. This study investigates how on-demand LLM-generated explanatory feedback influences learning in seven scenario-based tutor training lessons. Analyzing over 2,600 lesson completions from 885 tutor learners, we compare posttest performance among learners across three groups: learners who received feedback generated by gpt-3.5-turbo, those who declined it, and those without access. All groups received non-LLM corrective feedback. To address potential selection bias-where higher-performing learners may be more inclined to use LLM feedback-we applied propensity scoring. Learners with a higher predicted likelihood of engaging with LLM feedback scored significantly higher at posttest than those with lower propensity. After adjusting for this effect, two out of seven lessons showed statistically significant learning benefits from LLM feedback with standardized effect sizes of 0.28 and 0.33. These moderate effects suggest that the effectiveness of LLM feedback depends on the learners' tendency to seek support. Importantly, LLM feedback did not significantly increase completion time, and learners overwhelmingly rated it as helpful. These findings highlight LLM feedback's potential as a low-cost and scalable way to improve learning on open-ended tasks, particularly in existing systems already providing feedback without LLMs. This work contributes open datasets, LLM prompts, and rubrics to support reproducibility.