# Giving Feedback on Interactive Student Programs with Meta-Exploration

arXiv: https://arxiv.org/abs/2211.08802 · HTML: https://mlsanity.com/paper/2211.08802/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 59 references that are in the map.
- First posted on arXiv 16 Nov 2022.

## Builds on (most cited first, 30 of 32)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning](https://mlsanity.com/paper/1910.10897/) (2019)
- [Dueling Network Architectures for Deep Reinforcement Learning](https://mlsanity.com/paper/1511.06581/) (2015)
- [RL2: Fast Reinforcement Learning via Slow Reinforcement Learning](https://mlsanity.com/paper/1611.02779/) (2016)
- [Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables](https://mlsanity.com/paper/1903.08254/) (2019)
- [Learning to reinforcement learn](https://mlsanity.com/paper/1611.05763/) (2016)
- [Deep Knowledge Tracing](https://mlsanity.com/paper/1506.05908/) (2015)
- [Learning to Adapt in Dynamic, Real-World Environments Through Meta-Reinforcement Learning](https://mlsanity.com/paper/1803.11347/) (2018)
- [VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning](https://mlsanity.com/paper/1910.08348/) (2019)
- [Meta-Reinforcement Learning of Structured Exploration Strategies](https://mlsanity.com/paper/1802.07245/) (2018)
- [ProMP: Proximal Meta-Policy Search](https://mlsanity.com/paper/1810.06784/) (2018)
- [Meta reinforcement learning as task inference](https://mlsanity.com/paper/1905.06424/) (2019)
- [Decoupling Exploration and Exploitation for Meta-Reinforcement Learning without Sacrifices](https://mlsanity.com/paper/2008.02790/) (2020)
- [Meta Reinforcement Learning for Sim-to-real Domain Adaptation](https://mlsanity.com/paper/1909.12906/) (2019)
- [Environment Probing Interaction Policies](https://mlsanity.com/paper/1907.11740/) (2019)
- [Alchemy: A benchmark and analysis toolkit for meta-reinforcement learning agents](https://mlsanity.com/paper/2102.02926/) (2021)
- [Meta-Reinforcement Learning for Robotic Industrial Insertion Tasks](https://mlsanity.com/paper/2004.14404/) (2020)
- [Improving Playtesting Coverage via Curiosity Driven Reinforcement Learning Agents](https://mlsanity.com/paper/2103.13798/) (2021)
- [An Imitation Learning Approach for Cache Replacement](https://mlsanity.com/paper/2006.16239/) (2020)
- [Learning Program Embeddings to Propagate Feedback on Student Code](https://mlsanity.com/paper/1505.05969/) (2015)
- [Zero Shot Learning for Code Education: Rubric Sampling with Deep Learning Inference](https://mlsanity.com/paper/1809.01357/) (2018)
- [Learning Adaptive Exploration Strategies in Dynamic Environments Through Informed Policy Regularization](https://mlsanity.com/paper/2005.02934/) (2020)
- [Automated Correction for Syntax Errors in Programming Assignments using Recurrent Neural Networks](https://mlsanity.com/paper/1603.06129/) (2016)
- [The Continuous Hint Factory - Providing Hints in Vast and Sparsely Populated Edit Distance Spaces](https://mlsanity.com/paper/1708.06564/) (2017)
- [ProtoTransformer: A Meta-Learning Approach to Providing Student Feedback](https://mlsanity.com/paper/2107.14035/) (2021)
- [Predicting Student Dropout in Higher Education](https://mlsanity.com/paper/1606.06364/) (2016)
- [MAME : Model-Agnostic Meta-Exploration](https://mlsanity.com/paper/1911.04024/) (2019)

## Built on by (most cited first, 6 of 6)

- [A Chain-of-Thought Prompting Approach with LLMs for Evaluating Students' Formative Assessment Responses in Science](https://mlsanity.com/paper/2403.14565/) (2024)
- [Off-Policy Evaluation for Human Feedback](https://mlsanity.com/paper/2310.07123/) (2023)
- [CoTAL: Human-in-the-Loop Prompt Engineering for Generalizable Formative Assessment Scoring and Feedback](https://mlsanity.com/paper/2504.02323/) (2025)
- [Generating Language Corrections for Teaching Physical Control Tasks](https://mlsanity.com/paper/2306.07012/) (2023)
- [Off-Policy Selection for Initiating Human-Centric Experimental Design](https://mlsanity.com/paper/2410.20017/) (2024)
- [Discovering High-Quality Chess Puzzles with Offline Reinforcement Learning](https://mlsanity.com/paper/2608.14851/) (2026)

## Abstract (arXiv)

Developing interactive software, such as websites or games, is a particularly engaging way to learn computer science. However, teaching and giving feedback on such software is time-consuming -- standard approaches require instructors to manually grade student-implemented interactive programs. As a result, online platforms that serve millions, like Code.org, are unable to provide any feedback on assignments for implementing interactive programs, which critically hinders students' ability to learn. One approach toward automatic grading is to learn an agent that interacts with a student's program and explores states indicative of errors via reinforcement learning. However, existing work on this approach only provides binary feedback of whether a program is correct or not, while students require finer-grained feedback on the specific errors in their programs to understand their mistakes. In this work, we show that exploring to discover errors can be cast as a meta-exploration problem. This enables us to construct a principled objective for discovering errors and an algorithm for optimizing this objective, which provides fine-grained feedback. We evaluate our approach on a set of over 700K real anonymized student programs from a Code.org interactive assignment. Our approach provides feedback with 94.3% accuracy, improving over existing approaches by 17.7% and coming within 1.5% of human-level accuracy. Project web page: https://ezliu.github.io/dreamgrader.