# Effects of Human vs. Automatic Feedback on Students' Understanding of AI Concepts and Programming Style

arXiv: https://arxiv.org/abs/2011.10653 · HTML: https://mlsanity.com/paper/2011.10653/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Nov 2020.

## Built on by (most cited first, 5 of 5)

- ["It's not like Jarvis, but it's pretty close!" -- Examining ChatGPT's Usage among Undergraduate Students in Computer Science](https://mlsanity.com/paper/2311.09651/) (2023)
- [LabelAId: Just-in-time AI Interventions for Improving Human Labeling Quality and Domain Knowledge in Crowdsourcing Systems](https://mlsanity.com/paper/2403.09810/) (2024)
- [Evaluating Trust in AI, Human, and Co-produced Feedback Among Undergraduate Students](https://mlsanity.com/paper/2504.10961/) (2025)
- [Data Therapist: Eliciting Domain Knowledge from Subject Matter Experts Using Large Language Models](https://mlsanity.com/paper/2505.00455/) (2025)
- [How Adding Metacognitive Requirements in Support of AI Feedback in Practice Exams Transforms Student Learning Behaviors](https://mlsanity.com/paper/2505.13381/) (2025)

## Abstract (arXiv)

The use of automatic grading tools has become nearly ubiquitous in large undergraduate programming courses, and recent work has focused on improving the quality of automatically generated feedback. However, there is a relative lack of data directly comparing student outcomes when receiving computer-generated feedback and human-written feedback. This paper addresses this gap by splitting one 90-student class into two feedback groups and analyzing differences in the two cohorts' performance. The class is an intro to AI with programming HW assignments. One group of students received detailed computer-generated feedback on their programming assignments describing which parts of the algorithms' logic was missing; the other group additionally received human-written feedback describing how their programs' syntax relates to issues with their logic, and qualitative (style) recommendations for improving their code. Results on quizzes and exam questions suggest that human feedback helps students obtain a better conceptual understanding, but analyses found no difference between the groups' ability to collaborate on the final project. The course grade distribution revealed that students who received human-written feedback performed better overall; this effect was the most pronounced in the middle two quartiles of each group. These results suggest that feedback about the syntax-logic relation may be a primary mechanism by which human feedback improves student outcomes.