# Zero Shot Learning for Code Education: Rubric Sampling with Deep Learning Inference

arXiv: https://arxiv.org/abs/1809.01357 · HTML: https://mlsanity.com/paper/1809.01357/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Sep 2018.

## Built on by (most cited first, 16 of 16)

- [Open Source Language Models Can Provide Feedback: Evaluating LLMs' Ability to Help Students Using GPT-4-As-A-Judge](https://mlsanity.com/paper/2405.05253/) (2024)
- [Simultaneous Missing Value Imputation and Structure Learning with Groups](https://mlsanity.com/paper/2110.08223/) (2021)
- [Neural Task Synthesis for Visual Programming](https://mlsanity.com/paper/2305.18342/) (2023)
- [Generative AI for Education (GAIED): Advances, Opportunities, and Challenges](https://mlsanity.com/paper/2402.01580/) (2024)
- [Adaptive Scaffolding in Block-Based Programming via Synthesizing New Tasks as Pop Quizzes](https://mlsanity.com/paper/2303.16359/) (2023)
- [ProtoTransformer: A Meta-Learning Approach to Providing Student Feedback](https://mlsanity.com/paper/2107.14035/) (2021)
- [From {Solution Synthesis} to {Student Attempt Synthesis} for Block-Based Visual Programming Tasks](https://mlsanity.com/paper/2205.01265/) (2022)
- [Giving Feedback on Interactive Student Programs with Meta-Exploration](https://mlsanity.com/paper/2211.08802/) (2022)
- [Proactive and Reactive Engagement of Artificial Intelligence Methods for Education: A Review](https://mlsanity.com/paper/2301.10231/) (2023)
- [Play to Grade: Testing Coding Games as Classifying Markov Decision Process](https://mlsanity.com/paper/2110.14615/) (2021)
- [ZC3: Zero-Shot Cross-Language Code Clone Detection](https://mlsanity.com/paper/2308.13754/) (2023)
- [Automating Program Structure Classification](https://mlsanity.com/paper/2101.10087/) (2021)
- [Lost in Translation: Reimagining the Machine Learning Life Cycle in Education](https://mlsanity.com/paper/2209.03929/) (2022)
- [Synthesizing a Progression of Subtasks for Block-Based Visual Programming Tasks](https://mlsanity.com/paper/2305.17518/) (2023)
- [Assessing the Code Clone Detection Capability of Large Language Models](https://mlsanity.com/paper/2407.02402/) (2024)
- [AI-Enabled grading with near-domain data for scaling feedback with human-level accuracy](https://mlsanity.com/paper/2512.04113/) (2025)

## Abstract (arXiv)

In modern computer science education, massive open online courses (MOOCs) log thousands of hours of data about how students solve coding challenges. Being so rich in data, these platforms have garnered the interest of the machine learning community, with many new algorithms attempting to autonomously provide feedback to help future students learn. But what about those first hundred thousand students? In most educational contexts (i.e. classrooms), assignments do not have enough historical data for supervised learning. In this paper, we introduce a human-in-the-loop "rubric sampling" approach to tackle the "zero shot" feedback challenge. We are able to provide autonomous feedback for the first students working on an introductory programming assignment with accuracy that substantially outperforms data-hungry algorithms and approaches human level fidelity. Rubric sampling requires minimal teacher effort, can associate feedback with specific parts of a student's solution and can articulate a student's misconceptions in the language of the instructor. Deep learning inference enables rubric sampling to further improve as more assignment specific student data is acquired. We demonstrate our results on a novel dataset from Code.org, the world's largest programming education platform.