# Seeing the Forest and the Trees: Solving Visual Graph and Tree Based Data Structure Problems using Large Multimodal Models

arXiv: https://arxiv.org/abs/2412.11088 · HTML: https://mlsanity.com/paper/2412.11088/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 22 of its 74 references that are in the map.
- First posted on arXiv 15 Dec 2024.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 22 of 22)

- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [Programming Is Hard -- Or at Least It Used to Be: Educational Opportunities And Challenges of AI Code Generation](https://mlsanity.com/paper/2212.01020/) (2022)
- [Three things everyone should know about Vision Transformers](https://mlsanity.com/paper/2203.09795/) (2022)
- [Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming](https://mlsanity.com/paper/2302.07427/) (2023)
- [The Robots are Here: Navigating the Generative AI Revolution in Computing Education](https://mlsanity.com/paper/2310.00658/) (2023)
- [Conversing with Copilot: Exploring Prompt Engineering for Solving CS1 Problems Using Natural Language](https://mlsanity.com/paper/2210.15157/) (2022)
- [CodeHelp: Using Large Language Models with Guardrails for Scalable Support in Programming Classes](https://mlsanity.com/paper/2308.06921/) (2023)
- [Comparing Code Explanations Created by Students and Large Language Models](https://mlsanity.com/paper/2304.03938/) (2023)
- [The Widening Gap: The Benefits and Harms of Generative AI for Novice Programmers](https://mlsanity.com/paper/2405.17739/) (2024)
- [Thrilled by Your Progress! Large Language Models (GPT-4) No Longer Struggle to Pass Assessments in Higher Education Programming Courses](https://mlsanity.com/paper/2306.10073/) (2023)
- [Large Language Models (GPT) Struggle to Answer Multiple-Choice Questions about Code](https://mlsanity.com/paper/2303.08033/) (2023)
- [Can GPT-4 Support Analysis of Textual Data in Tasks Requiring Highly Specialized Domain Expertise?](https://mlsanity.com/paper/2306.13906/) (2023)
- [The Effects of Generative AI on Computing Students' Help-Seeking Preferences](https://mlsanity.com/paper/2401.02262/) (2024)
- [Promptly: Using Prompt Problems to Teach Learners How to Effectively Utilize AI Code Generators](https://mlsanity.com/paper/2307.16364/) (2023)
- [Desirable Characteristics for AI Teaching Assistants in Programming Education](https://mlsanity.com/paper/2405.14178/) (2024)
- [ChatGPT, Can You Generate Solutions for my Coding Exercises? An Evaluation on its Effectiveness in an undergraduate Java Programming Course](https://mlsanity.com/paper/2305.13680/) (2023)
- [More Robots are Coming: Large Multimodal Models (ChatGPT) can Solve Visually Diverse Images of Parsons Problems](https://mlsanity.com/paper/2311.04926/) (2023)
- ["It's not like Jarvis, but it's pretty close!" -- Examining ChatGPT's Usage among Undergraduate Students in Computer Science](https://mlsanity.com/paper/2311.09651/) (2023)
- ["Like a Nesting Doll": Analyzing Recursion Analogies Generated by CS Students using Large Language Models](https://mlsanity.com/paper/2403.09409/) (2024)
- [Chalktalk : A Visualization and Communication Language -- As a Tool in the Domain of Computer Science Education](https://mlsanity.com/paper/1809.07166/) (2018)
- [Imagining Computing Education Assessment after Generative AI](https://mlsanity.com/paper/2401.04601/) (2024)
- [Can ChatGPT Pass a Theory of Computing Course?](https://mlsanity.com/paper/2407.07757/) (2024)

## Built on by (most cited first, 2 of 2)

- [Exploring Student Behaviors and Motivations using AI TAs with Optional Guardrails](https://mlsanity.com/paper/2504.11146/) (2025)
- [Exploring Student Choice and the Use of Multimodal Generative AI in Programming Learning](https://mlsanity.com/paper/2510.05417/) (2025)

## Abstract (arXiv)

Recent advancements in generative AI systems have raised concerns about academic integrity among educators. Beyond excelling at solving programming problems and text-based multiple-choice questions, recent research has also found that large multimodal models (LMMs) can solve Parsons problems based only on an image. However, such problems are still inherently text-based and rely on the capabilities of the models to convert the images of code blocks to their corresponding text. In this paper, we further investigate the capabilities of LMMs to solve graph and tree data structure problems based only on images. To achieve this, we computationally construct and evaluate a novel benchmark dataset comprising 9,072 samples of diverse graph and tree data structure tasks to assess the performance of the GPT-4o, GPT-4v, Gemini 1.5 Pro, Gemini 1.5 Flash, Gemini 1.0 Pro Vision, and Claude 3 model families. GPT-4o and Gemini 1.5 Flash performed best on trees and graphs respectively. GPT-4o achieved 87.6% accuracy on tree samples, while Gemini 1.5 Flash, achieved 56.2% accuracy on graph samples. Our findings highlight the influence of structural and visual variations on model performance. This research not only introduces an LMM benchmark to facilitate replication and further exploration but also underscores the potential of LMMs in solving complex computing problems, with important implications for pedagogy and assessment practices.