# NERIF: GPT-4V for Automatic Scoring of Drawn Models arXiv: https://arxiv.org/abs/2311.12990 ยท HTML: https://mlsanity.com/paper/2311.12990/ ## Key facts - Cited by 13 papers in the map (data through 14 Oct 2026). - Builds on 19 of its 61 references that are in the map. - First posted on arXiv 21 Nov 2023. ## Builds on (most cited first, 19 of 19) - [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023) - [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022) - [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015) - [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015) - [Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting](https://mlsanity.com/paper/2305.04388/) (2023) - [The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)](https://mlsanity.com/paper/2309.17421/) (2023) - [Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models](https://mlsanity.com/paper/2303.04671/) (2023) - [Generalizing from a Few Examples: A Survey on Few-Shot Learning](https://mlsanity.com/paper/1904.05046/) (2019) - [Role-Play with Large Language Models](https://mlsanity.com/paper/2305.16367/) (2023) - [Can GPT-4V(ision) Serve Medical Applications? Case Studies on GPT-4V for Multimodal Medical Diagnosis](https://mlsanity.com/paper/2310.09909/) (2023) - [Ablation Studies in Artificial Neural Networks](https://mlsanity.com/paper/1901.08644/) (2019) - [Towards End-to-End Embodied Decision Making via Multi-modal Large Language Model: Explorations with GPT4-Vision and Beyond](https://mlsanity.com/paper/2310.02071/) (2023) - [Fine-tuning ChatGPT for Automatic Scoring](https://mlsanity.com/paper/2310.10072/) (2023) - [A Review on Explainability in Multimodal Deep Neural Nets](https://mlsanity.com/paper/2105.07878/) (2021) - [Can Language Models be Instructed to Protect Personal Information?](https://mlsanity.com/paper/2310.02224/) (2023) - [Exploring OCR Capabilities of GPT-4V(ision) : A Quantitative and In-depth Evaluation](https://mlsanity.com/paper/2310.16809/) (2023) - [An Early Evaluation of GPT-4V(ision)](https://mlsanity.com/paper/2310.16534/) (2023) - [Using GPT-4 to Augment Unbalanced Data for Automatic Scoring](https://mlsanity.com/paper/2310.18365/) (2023) - [GPT-4 Vision on Medical Image Classification -- A Case Study on COVID-19 Dataset](https://mlsanity.com/paper/2310.18498/) (2023) ## Built on by (most cited first, 13 of 13) - [A Systematic Assessment of OpenAI o1-Preview for Higher Order Thinking in Education](https://mlsanity.com/paper/2410.21287/) (2024) - [Taking the Next Step with Generative Artificial Intelligence: The Transformative Role of Multimodal Large Language Models in Science Education](https://mlsanity.com/paper/2401.00832/) (2024) - [Gemini Pro Defeated by GPT-4V: Evidence from Education](https://mlsanity.com/paper/2401.08660/) (2023) - [LLaVA-Docent: Instruction Tuning with Multimodal Large Language Model to Support Art Appreciation Education](https://mlsanity.com/paper/2402.06264/) (2024) - [Transforming Teachers' Roles and Agencies in the Era of Generative AI: Perceptions, Acceptance, Knowledge, and Practices](https://mlsanity.com/paper/2410.03018/) (2024) - [Can generative AI and ChatGPT outperform humans on cognitive-demanding problem-solving tasks in science?](https://mlsanity.com/paper/2401.15081/) (2024) - [Realizing Visual Question Answering for Education: GPT-4V as a Multimodal AI](https://mlsanity.com/paper/2405.07163/) (2024) - [SketchMind: A Multi-Agent Cognitive Framework for Assessing Student-Drawn Scientific Sketches](https://mlsanity.com/paper/2507.22904/) (2025) - [An Early Investigation into the Utility of Multimodal Large Language Models in Medical Imaging](https://mlsanity.com/paper/2406.00667/) (2024) - [Can MLLMs generate human-like feedback in grading multimodal short answers?](https://mlsanity.com/paper/2412.19755/) (2024) - [LLM-as-a-judge validity is strongly task-dependent across physics assessment formats](https://mlsanity.com/paper/2603.14732/) (2026) - [Captioning Daily Activity Images in Early Childhood Education: Benchmark and Algorithm](https://mlsanity.com/paper/2604.01941/) (2026) - [Confidence-Aware Automated Assessment of Student-Drawn Scientific Models](https://mlsanity.com/paper/2606.20264/) (2026) ## Abstract (arXiv) Scoring student-drawn models is time-consuming. Recently released GPT-4V provides a unique opportunity to advance scientific modeling practices by leveraging the powerful image processing capability. To test this ability specifically for automatic scoring, we developed a method NERIF (Notation-Enhanced Rubric Instruction for Few-shot Learning) employing instructional note and rubrics to prompt GPT-4V to score students' drawn models for science phenomena. We randomly selected a set of balanced data (N = 900) that includes student-drawn models for six modeling assessment tasks. Each model received a score from GPT-4V ranging at three levels: 'Beginning,' 'Developing,' or 'Proficient' according to scoring rubrics. GPT-4V scores were compared with human experts' scores to calculate scoring accuracy. Results show that GPT-4V's average scoring accuracy was mean =.51, SD = .037. Specifically, average scoring accuracy was .64 for the 'Beginning' class, .62 for the 'Developing' class, and .26 for the 'Proficient' class, indicating that more proficient models are more challenging to score. Further qualitative study reveals how GPT-4V retrieves information from image input, including problem context, example evaluations provided by human coders, and students' drawing models. We also uncovered how GPT-4V catches the characteristics of student-drawn models and narrates them in natural language. At last, we demonstrated how GPT-4V assigns scores to student-drawn models according to the given scoring rubric and instructional notes. Our findings suggest that the NERIF is an effective approach for employing GPT-4V to score drawn models. Even though there is space for GPT-4V to improve scoring accuracy, some mis-assigned scores seemed interpretable to experts. The results of this study show that utilizing GPT-4V for automatic scoring of student-drawn models is promising.