# Realizing Visual Question Answering for Education: GPT-4V as a Multimodal AI arXiv: https://arxiv.org/abs/2405.07163 ยท HTML: https://mlsanity.com/paper/2405.07163/ ## Key facts - Cited by 5 papers in the map (data through 14 Oct 2026). - Builds on 20 of its 40 references that are in the map. - First posted on arXiv 12 May 2024. ## Builds on (most cited first, 20 of 20) - [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021) - [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023) - [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022) - [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021) - [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015) - [VisualBERT: A Simple and Performant Baseline for Vision and Language](https://mlsanity.com/paper/1908.03557/) (2019) - [Vision-Language Models for Vision Tasks: A Survey](https://mlsanity.com/paper/2304.00685/) (2023) - [The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)](https://mlsanity.com/paper/2309.17421/) (2023) - [Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models](https://mlsanity.com/paper/2303.04671/) (2023) - [Can GPT-4V(ision) Serve Medical Applications? Case Studies on GPT-4V for Multimodal Medical Diagnosis](https://mlsanity.com/paper/2310.09909/) (2023) - [Applying Large Language Models and Chain-of-Thought for Automatic Scoring](https://mlsanity.com/paper/2312.03748/) (2023) - [The AGI Containment Problem](https://mlsanity.com/paper/1604.00545/) (2016) - [An Early Evaluation of GPT-4V(ision)](https://mlsanity.com/paper/2310.16534/) (2023) - [From Image to Language: A Critical Analysis of Visual Question Answering (VQA) Approaches, Challenges, and Opportunities](https://mlsanity.com/paper/2311.00308/) (2023) - [Multimodality of AI for Education: Towards Artificial General Intelligence](https://mlsanity.com/paper/2312.06037/) (2023) - [Using GPT-4 to Augment Unbalanced Data for Automatic Scoring](https://mlsanity.com/paper/2310.18365/) (2023) - [Taking the Next Step with Generative Artificial Intelligence: The Transformative Role of Multimodal Large Language Models in Science Education](https://mlsanity.com/paper/2401.00832/) (2024) - [NERIF: GPT-4V for Automatic Scoring of Drawn Models](https://mlsanity.com/paper/2311.12990/) (2023) - [PhysicsAssistant: An LLM-Powered Interactive Learning Robot for Physics Lab Investigations](https://mlsanity.com/paper/2403.18721/) (2024) - [Collaborative Learning with Artificial Intelligence Speakers (CLAIS): Pre-Service Elementary Science Teachers' Responses to the Prototype](https://mlsanity.com/paper/2401.05400/) (2023) ## Built on by (most cited first, 5 of 5) - [Transforming Teachers' Roles and Agencies in the Era of Generative AI: Perceptions, Acceptance, Knowledge, and Practices](https://mlsanity.com/paper/2410.03018/) (2024) - [Evaluating Vision-Language and Large Language Models for Automated Student Assessment in Indonesian Classrooms](https://mlsanity.com/paper/2506.04822/) (2025) - [Build AI Assistants using Large Language Models and Agents to Enhance the Engineering Education of Biomechanics](https://mlsanity.com/paper/2511.15752/) (2025) - [VQA support to Arabic Language Learning Educational Tool](https://mlsanity.com/paper/2508.03488/) (2025) - [Can Vision Language Models Be Adaptive in Mathematics Education? A Learner Model-based Rubric Study](https://mlsanity.com/paper/2605.16011/) (2026) ## Abstract (arXiv) Educational scholars have analyzed various image data acquired from teaching and learning situations, such as photos that shows classroom dynamics, students' drawings with regard to the learning content, textbook illustrations, etc. Unquestioningly, most qualitative analysis of and explanation on image data have been conducted by human researchers, without machine-based automation. It was partially because most image processing artificial intelligence models were not accessible to general educational scholars or explainable due to their complex deep neural network architecture. However, the recent development of Visual Question Answering (VQA) techniques is accomplishing usable visual language models, which receive from the user a question about the given image and returns an answer, both in natural language. Particularly, GPT-4V released by OpenAI, has wide opened the state-of-the-art visual langauge model service so that VQA could be used for a variety of purposes. However, VQA and GPT-4V have not yet been applied to educational studies much. In this position paper, we suggest that GPT-4V contributes to realizing VQA for education. By 'realizing' VQA, we denote two meanings: (1) GPT-4V realizes the utilization of VQA techniques by any educational scholars without technical/accessibility barrier, and (2) GPT-4V makes educational scholars realize the usefulness of VQA to educational research. Given these, this paper aims to introduce VQA for educational studies so that it provides a milestone for educational research methodology. In this paper, chapter II reviews the development of VQA techniques, which primes with the release of GPT-4V. Chapter III reviews the use of image analysis in educational studies. Chapter IV demonstrates how GPT-4V can be used for each research usage reviewed in Chapter III, with operating prompts provided. Finally, chapter V discusses the future implications.