# Play to Grade: Testing Coding Games as Classifying Markov Decision Process

arXiv: https://arxiv.org/abs/2110.14615 · HTML: https://mlsanity.com/paper/2110.14615/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 51 references that are in the map.
- First posted on arXiv 27 Oct 2021.

## Builds on (most cited first, 19 of 19)

- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Curiosity-driven Exploration by Self-supervised Prediction](https://mlsanity.com/paper/1705.05363/) (2017)
- [Pixel Recurrent Neural Networks](https://mlsanity.com/paper/1601.06759/) (2016)
- [Diversity is All You Need: Learning Skills without a Reward Function](https://mlsanity.com/paper/1802.06070/) (2018)
- [Deep Knowledge Tracing](https://mlsanity.com/paper/1506.05908/) (2015)
- [Go-Explore: a New Approach for Hard-Exploration Problems](https://mlsanity.com/paper/1901.10995/) (2019)
- [Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning](https://mlsanity.com/paper/1509.08731/) (2015)
- [Trial without Error: Towards Safe Reinforcement Learning via Human Intervention](https://mlsanity.com/paper/1707.05173/) (2017)
- [Plannable Approximations to MDP Homomorphisms: Equivariance under Actions](https://mlsanity.com/paper/2002.11963/) (2020)
- [Learning Human Objectives by Evaluating Hypothetical Behavior](https://mlsanity.com/paper/1912.05652/) (2019)
- [Improving Playtesting Coverage via Curiosity Driven Reinforcement Learning Agents](https://mlsanity.com/paper/2103.13798/) (2021)
- [Learning Program Embeddings to Propagate Feedback on Student Code](https://mlsanity.com/paper/1505.05969/) (2015)
- [Zero Shot Learning for Code Education: Rubric Sampling with Deep Learning Inference](https://mlsanity.com/paper/1809.01357/) (2018)
- [Automated Correction for Syntax Errors in Programming Assignments using Recurrent Neural Networks](https://mlsanity.com/paper/1603.06129/) (2016)
- [The Continuous Hint Factory - Providing Hints in Vast and Sparsely Populated Edit Distance Spaces](https://mlsanity.com/paper/1708.06564/) (2017)
- [Automated Essay Scoring based on Two-Stage Learning](https://mlsanity.com/paper/1901.07744/) (2019)
- [Improving Safety in Reinforcement Learning Using Model-Based Architectures and Human Intervention](https://mlsanity.com/paper/1903.09328/) (2019)
- [Taking the Scenic Route: Automatic Exploration for Videogames](https://mlsanity.com/paper/1812.03125/) (2018)
- [Reliable Deep Grade Prediction with Uncertainty Estimation](https://mlsanity.com/paper/1902.10213/) (2019)

## Built on by (most cited first, 3 of 3)

- [Generative AI for Education (GAIED): Advances, Opportunities, and Challenges](https://mlsanity.com/paper/2402.01580/) (2024)
- [Giving Feedback on Interactive Student Programs with Meta-Exploration](https://mlsanity.com/paper/2211.08802/) (2022)
- [Robust Phi-Divergence MDPs](https://mlsanity.com/paper/2205.14202/) (2022)

## Abstract (arXiv)

Contemporary coding education often presents students with the task of developing programs that have user interaction and complex dynamic systems, such as mouse based games. While pedagogically compelling, there are no contemporary autonomous methods for providing feedback. Notably, interactive programs are impossible to grade by traditional unit tests. In this paper we formalize the challenge of providing feedback to interactive programs as a task of classifying Markov Decision Processes (MDPs). Each student's program fully specifies an MDP where the agent needs to operate and decide, under reasonable generalization, if the dynamics and reward model of the input MDP should be categorized as correct or broken. We demonstrate that by designing a cooperative objective between an agent and an autoregressive model, we can use the agent to sample differential trajectories from the input MDP that allows a classifier to determine membership: Play to Grade. Our method enables an automatic feedback system for interactive code assignments. We release a dataset of 711,274 anonymized student submissions to a single assignment with hand-coded bug labels to support future research.