# A Dataset for Modeling Iterative Problem-Solving

arXiv: https://arxiv.org/abs/2609.00940 · HTML: https://mlsanity.com/paper/2609.00940/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 57 references that are in the map.
- First posted on arXiv 1 Sep 2026.

## Builds on (most cited first, 16 of 16)

- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models](https://mlsanity.com/paper/2506.05176/) (2025)
- [Mastering Atari with Discrete World Models](https://mlsanity.com/paper/2010.02193/) (2020)
- [FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance](https://mlsanity.com/paper/2305.05176/) (2023)
- [RouteLLM: Learning to Route LLMs with Preference Data](https://mlsanity.com/paper/2406.18665/) (2024)
- [Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies](https://mlsanity.com/paper/2208.10264/) (2022)
- [Deep Knowledge Tracing](https://mlsanity.com/paper/1506.05908/) (2015)
- [A Survey of Knowledge Tracing: Models, Variants, and Applications](https://mlsanity.com/paper/2105.15106/) (2021)
- [SMART: Simulated Students Aligned with Item Response Theory for Question Difficulty Prediction](https://mlsanity.com/paper/2507.05129/) (2025)
- [Towards Modeling Learner Performance with Large Language Models](https://mlsanity.com/paper/2403.14661/) (2024)
- [Learning to Make MISTAKEs: Modeling Incorrect Student Thinking And Key Errors](https://mlsanity.com/paper/2510.11502/) (2025)
- [Code-DKT: A Code-based Knowledge Tracing Model for Programming Tasks](https://mlsanity.com/paper/2206.03545/) (2022)
- [Unbounded Human Learning: Optimal Scheduling for Spaced Repetition](https://mlsanity.com/paper/1602.07032/) (2016)
- [Test Case-Informed Knowledge Tracing for Open-ended Coding Tasks](https://mlsanity.com/paper/2410.10829/) (2024)
- [FGIT: Fault-Guided Fine-Tuning for Code Generation](https://mlsanity.com/paper/2503.16913/) (2025)

## Abstract (arXiv)

Solving problems through repeated attempts is a sequential modeling task: at each step, the solver receives feedback and decides how to revise their solutions. Predicting whether performance improves, plateaus, or regresses across attempts is central to understanding any iterative problem-solving process in both human learners and autonomous agents. Beyond outcomes, modeling what errors persist and how strategies shift across attempts provides deeper insight into the mechanics of sequential learning. Studying these dynamics requires observing many solvers as they attempt, receive feedback, and revise. Programming courses with automated grading provide this setting, as students iteratively submit code to test suites and receive feedback on every attempt. We therefore curate CodeInsight, a large-scale dataset of over 3 million submissions from 3,286 undergraduates across 2 introductory C++ courses in 2 academic years, with test-case-level outcomes, timestamps, and source code. On this dataset, we build a benchmark that evaluates models spanning parametric, sequential, and generative traditions under a shared calibration-and-scoring protocol, including a Recurrent State Space Model (RSSM) adapted to track solver characteristics through discrete latent variables and an LLM-based predictor that generates explicit solutions. The adapted RSSM achieves the strongest predictive accuracy on three of the four courses. The LLM predictor is less accurate but produces full submissions at each attempt, enabling direct analysis of failure modes. We find that the model's coding proficiency is inversely related to predictive performance in this setting, with the LLM better understood as a generative solver conditioned on context rather than a faithful predictor of solver behavior. We publicly release our code and the dataset on request to facilitate future research.