# Can MLLMs generate human-like feedback in grading multimodal short answers?

arXiv: https://arxiv.org/abs/2412.19755 · HTML: https://mlsanity.com/paper/2412.19755/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 23 references that are in the map.
- First posted on arXiv 27 Dec 2024.

## Builds on (most cited first, 9 of 9)

- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions](https://mlsanity.com/paper/2311.05232/) (2023)
- [ROUGE 2.0: Updated and Improved Measures for Evaluation of Summarization Tasks](https://mlsanity.com/paper/1803.01937/) (2018)
- [Distilling ChatGPT for Explainable Automated Student Answer Assessment](https://mlsanity.com/paper/2305.12962/) (2023)
- [NERIF: GPT-4V for Automatic Scoring of Drawn Models](https://mlsanity.com/paper/2311.12990/) (2023)
- [Pre-training Language Models for Comparative Reasoning](https://mlsanity.com/paper/2305.14457/) (2023)
- [Beyond Scores: A Modular RAG-Based System for Automatic Short Answer Scoring with Feedback](https://mlsanity.com/paper/2409.20042/) (2024)
- [Knowledge Distillation Transfer Sets and their Impact on Downstream NLU Tasks](https://mlsanity.com/paper/2210.04834/) (2022)

## Built on by (most cited first, 1 of 1)

- [Towards Transparent AI Grading: Semantic Entropy as a Signal for Human-AI Disagreement](https://mlsanity.com/paper/2508.04105/) (2025)

## Abstract (arXiv)

In education, the traditional Automatic Short Answer Grading (ASAG) with feedback problem has focused primarily on evaluating text-only responses. However, real-world assessments often include multimodal responses containing both diagrams and text. To address this limitation, we introduce the Multimodal Short Answer Grading with Feedback (MMSAF) problem, which requires jointly evaluating textual and diagrammatic content while also providing explanatory feedback. Collecting data representative of such multimodal responses is challenging due to both scale and logistical constraints. To mitigate this, we develop an automated data generation framework that leverages LLM hallucinations to mimic common student errors, thereby constructing a dataset of 2,197 instances. We evaluate 4 Multimodal Large Language Models (MLLMs) across 3 STEM subjects, showing that MLLMs achieve accuracies of up to 62.5% in predicting answer correctness (correct/partially correct/incorrect) and up to 80.36% in assessing image relevance. This also includes a human evaluation with 9 annotators across 5 parameters, including a rubric-based approach. The rubrics also serve as a way to evaluate the feedback quality semantically rather than using overlap-based approaches. Our findings highlight which MLLMs are better suited for such tasks while also pointing out to drawbacks of the remaining MLLMs.