# Can Large Language Models Replicate ITS Feedback on Open-Ended Math Questions?

arXiv: https://arxiv.org/abs/2405.06414 · HTML: https://mlsanity.com/paper/2405.06414/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 23 references that are in the map.
- First posted on arXiv 10 May 2024.

## Builds on (most cited first, 8 of 8)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [What Makes Good In-Context Examples for GPT-3?](https://mlsanity.com/paper/2101.06804/) (2021)
- [Automated Distractor and Feedback Generation for Math Multiple-choice Questions via In-context Learning](https://mlsanity.com/paper/2308.03234/) (2023)
- [Mathematical Language Processing: Automatic Grading and Feedback for Open Response Mathematical Questions](https://mlsanity.com/paper/1501.04346/) (2015)
- [Evaluating ChatGPT's Decimal Skills and Feedback Generation in a Digital Learning Game](https://mlsanity.com/paper/2306.16639/) (2023)
- [Automated Personalized Feedback Improves Learning Gains in an Intelligent Tutoring System](https://mlsanity.com/paper/2005.02431/) (2020)

## Built on by (most cited first, 4 of 4)

- [DiVERT: Distractor Generation with Variational Errors Represented as Text for Math Multiple-choice Questions](https://mlsanity.com/paper/2406.19356/) (2024)
- [Simulating Student-Tutor Interactions with LLMs for Multilingual Math Feedback](https://mlsanity.com/paper/2506.04920/) (2025)
- [Exploring LLM-Generated Feedback for Economics Essays: How Teaching Assistants Evaluate and Envision Its Use](https://mlsanity.com/paper/2505.15596/) (2025)
- [Gumbel Machine: Counterfactual Student Writing Generation via Gumbel Noise Steering](https://mlsanity.com/paper/2605.27249/) (2026)

## Abstract (arXiv)

Intelligent Tutoring Systems (ITSs) often contain an automated feedback component, which provides a predefined feedback message to students when they detect a predefined error. To such a feedback component, we often resort to template-based approaches. These approaches require significant effort from human experts to detect a limited number of possible student errors and provide corresponding feedback. This limitation is exemplified in open-ended math questions, where there can be a large number of different incorrect errors. In our work, we examine the capabilities of large language models (LLMs) to generate feedback for open-ended math questions, similar to that of an established ITS that uses a template-based approach. We fine-tune both open-source and proprietary LLMs on real student responses and corresponding ITS-provided feedback. We measure the quality of the generated feedback using text similarity metrics. We find that open-source and proprietary models both show promise in replicating the feedback they see during training, but do not generalize well to previously unseen student errors. These results suggest that despite being able to learn the formatting of feedback, LLMs are not able to fully understand mathematical errors made by students.