# Open Source Language Models Can Provide Feedback: Evaluating LLMs' Ability to Help Students Using GPT-4-As-A-Judge

arXiv: https://arxiv.org/abs/2405.05253 · HTML: https://mlsanity.com/paper/2405.05253/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 49 references that are in the map.
- First posted on arXiv 8 May 2024.

## Builds on (most cited first, 28 of 28)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [LIMA: Less Is More for Alignment](https://mlsanity.com/paper/2305.11206/) (2023)
- [Measuring Coding Challenge Competence With APPS](https://mlsanity.com/paper/2105.09938/) (2021)
- [Prometheus: Inducing Fine-grained Evaluation Capability in Language Models](https://mlsanity.com/paper/2310.08491/) (2023)
- [JudgeLM: Fine-tuned Large Language Models are Scalable Judges](https://mlsanity.com/paper/2310.17631/) (2023)
- [PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization](https://mlsanity.com/paper/2306.05087/) (2023)
- [Is ChatGPT better than Human Annotators? Potential and Limitations of ChatGPT in Explaining Implicit Hate Speech](https://mlsanity.com/paper/2302.07736/) (2023)
- [Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming](https://mlsanity.com/paper/2302.07427/) (2023)
- [The Robots are Here: Navigating the Generative AI Revolution in Computing Education](https://mlsanity.com/paper/2310.00658/) (2023)
- [Practical and Ethical Challenges of Large Language Models in Education: A Systematic Scoping Review](https://mlsanity.com/paper/2303.13379/) (2023)
- [Conversing with Copilot: Exploring Prompt Engineering for Solving CS1 Problems Using Natural Language](https://mlsanity.com/paper/2210.15157/) (2022)
- [CodeHelp: Using Large Language Models with Guardrails for Scalable Support in Programming Classes](https://mlsanity.com/paper/2308.06921/) (2023)
- [Using Large Language Models to Enhance Programming Error Messages](https://mlsanity.com/paper/2210.11630/) (2022)
- [Computing Education in the Era of Generative AI](https://mlsanity.com/paper/2306.02608/) (2023)
- [Thrilled by Your Progress! Large Language Models (GPT-4) No Longer Struggle to Pass Assessments in Higher Education Programming Courses](https://mlsanity.com/paper/2306.10073/) (2023)
- [Generative AI for Programming Education: Benchmarking ChatGPT, GPT-4, and Human Tutors](https://mlsanity.com/paper/2306.17156/) (2023)
- [Generating High-Precision Feedback for Programming Syntax Errors using Large Language Models](https://mlsanity.com/paper/2302.04662/) (2023)
- [Large Language Models (GPT) for automating feedback on programming assignments](https://mlsanity.com/paper/2307.00150/) (2023)
- [Prompt Problems: A New Programming Exercise for the Generative AI Era](https://mlsanity.com/paper/2311.05943/) (2023)
- [Automating Human Tutor-Style Programming Feedback: Leveraging GPT-4 Tutor Model for Hint Generation and GPT-3.5 Student Model for Hint Validation](https://mlsanity.com/paper/2310.03780/) (2023)
- [Exploring the Potential of Large Language Models to Generate Formative Programming Feedback](https://mlsanity.com/paper/2309.00029/) (2023)
- [Assessing the Quality of Multiple-Choice Questions Using GPT-4 and Rule-Based Methods](https://mlsanity.com/paper/2307.08161/) (2023)
- [Automated Distractor and Feedback Generation for Math Multiple-choice Questions via In-context Learning](https://mlsanity.com/paper/2308.03234/) (2023)
- [AI-TA: Towards an Intelligent Question-Answer Teaching Assistant using Open-Source LLMs](https://mlsanity.com/paper/2311.02775/) (2023)
- [Zero Shot Learning for Code Education: Rubric Sampling with Deep Learning Inference](https://mlsanity.com/paper/1809.01357/) (2018)

## Built on by (most cited first, 20 of 20)

- [Towards Implicit Bias Detection and Mitigation in Multi-Agent LLM Interactions](https://mlsanity.com/paper/2410.02584/) (2024)
- [TrustJudge: Inconsistencies of LLM-as-a-Judge and How to Alleviate Them](https://mlsanity.com/paper/2509.21117/) (2025)
- [ACE: A LLM-based Negotiation Coaching System](https://mlsanity.com/paper/2410.01555/) (2024)
- [Conversational User-AI Intervention: A Study on Prompt Rewriting for Improved LLM Response Generation](https://mlsanity.com/paper/2503.16789/) (2025)
- [Evaluating Language Models for Generating and Judging Programming Feedback](https://mlsanity.com/paper/2407.04873/) (2024)
- [The Value of Variance: Mitigating Debate Collapse in Multi-Agent Systems via Uncertainty-Driven Policy Optimization](https://mlsanity.com/paper/2602.07186/) (2026)
- [Towards Adaptive Feedback with AI: Comparing the Feedback Quality of LLMs and Teachers on Experimentation Protocols](https://mlsanity.com/paper/2502.12842/) (2025)
- [From First Draft to Final Insight: A Multi-Agent Approach for Feedback Generation](https://mlsanity.com/paper/2505.04869/) (2025)
- [Exploring Student Behaviors and Motivations using AI TAs with Optional Guardrails](https://mlsanity.com/paper/2504.11146/) (2025)
- [Breaking the Programming Language Barrier: Multilingual Prompting to Empower Non-Native English Learners](https://mlsanity.com/paper/2412.12800/) (2024)
- [Group Preference Alignment: Customized LLM Response Generation from In-Situ Conversations](https://mlsanity.com/paper/2503.08035/) (2025)
- [What Drives Students' Use of AI Chatbots? Technology Acceptance in Conversational AI](https://mlsanity.com/paper/2602.20547/) (2026)
- [A Knowledge-Component-Based Methodology for Evaluating AI Assistants](https://mlsanity.com/paper/2406.05603/) (2024)
- [On the Opportunities of Large Language Models for Programming Process Data](https://mlsanity.com/paper/2411.00414/) (2024)
- [Can LLM Assist in the Evaluation of the Quality of Machine Learning Explanations?](https://mlsanity.com/paper/2502.20635/) (2025)
- [WHERE and WHICH: Iterative Debate for Biomedical Synthetic Data Augmentation](https://mlsanity.com/paper/2503.23673/) (2025)
- [Narrowing the Gap: Supervised Fine-Tuning of Open-Source LLMs as a Viable Alternative to Proprietary Models for Pedagogical Tools](https://mlsanity.com/paper/2507.05305/) (2025)
- [Enabling Multi-Agent Systems as Learning Designers: Applying Learning Sciences to AI Instructional Design](https://mlsanity.com/paper/2508.16659/) (2025)
- [GEM: Generative Entropy-Guided Preference Modeling for Few-shot Alignment of LLMs](https://mlsanity.com/paper/2511.13007/) (2025)
- [An Explainable AI Assistant for Introductory Programming Education: Improving Feedback Reliability with Instructor-AI Collaboration](https://mlsanity.com/paper/2606.12425/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have shown great potential for the automatic generation of feedback in a wide range of computing contexts. However, concerns have been voiced around the privacy and ethical implications of sending student work to proprietary models. This has sparked considerable interest in the use of open source LLMs in education, but the quality of the feedback that such open models can produce remains understudied. This is a concern as providing flawed or misleading generated feedback could be detrimental to student learning. Inspired by recent work that has utilised very powerful LLMs, such as GPT-4, to evaluate the outputs produced by less powerful models, we conduct an automated analysis of the quality of the feedback produced by several open source models using a dataset from an introductory programming course. First, we investigate the viability of employing GPT-4 as an automated evaluator by comparing its evaluations with those of a human expert. We observe that GPT-4 demonstrates a bias toward positively rating feedback while exhibiting moderate agreement with human raters, showcasing its potential as a feedback evaluator. Second, we explore the quality of feedback generated by several leading open-source LLMs by using GPT-4 to evaluate the feedback. We find that some models offer competitive performance with popular proprietary LLMs, such as ChatGPT, indicating opportunities for their responsible use in educational settings.