# Multimodal LLMs Struggle with Basic Visual Network Analysis: a VNA Benchmark

arXiv: https://arxiv.org/abs/2405.06634 · HTML: https://mlsanity.com/paper/2405.06634/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 9 of its 17 references that are in the map.
- First posted on arXiv 10 May 2024.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 9 of 9)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Vision-Language Models for Vision Tasks: A Survey](https://mlsanity.com/paper/2304.00685/) (2023)
- [Exploring the Potential of Large Language Models (LLMs) in Learning on Graphs](https://mlsanity.com/paper/2307.03393/) (2023)
- [Talk like a Graph: Encoding Graphs for Large Language Models](https://mlsanity.com/paper/2310.04560/) (2023)
- [Good at captioning, bad at counting: Benchmarking GPT-4V on Earth observation data](https://mlsanity.com/paper/2401.17600/) (2024)
- [VisionGraph: Leveraging Large Multimodal Models for Graph Theory Problems in Visual Context](https://mlsanity.com/paper/2405.04950/) (2024)
- [Large Language Models for Social Networks: Applications, Challenges, and Solutions](https://mlsanity.com/paper/2401.02575/) (2024)
- [Zero-Shot Object Counting with Language-Vision Models](https://mlsanity.com/paper/2309.13097/) (2023)
- [Pushing Boundaries: Exploring Zero Shot Object Classification with Large Multimodal Models](https://mlsanity.com/paper/2401.00127/) (2023)

## Built on by (most cited first, 2 of 2)

- [VisGraphVar: A Benchmark Generator for Assessing Variability in Graph Analysis Using Large Vision-Language Models](https://mlsanity.com/paper/2411.14832/) (2024)
- [When Vision Meets Graphs: A Survey on Graph Reasoning and Learning](https://mlsanity.com/paper/2609.03816/) (2026)

## Abstract (arXiv)

We evaluate the zero-shot ability of GPT-4 and LLaVa to perform simple Visual Network Analysis (VNA) tasks on small-scale graphs. We evaluate the Vision Language Models (VLMs) on 5 tasks related to three foundational network science concepts: identifying nodes of maximal degree on a rendered graph, identifying whether signed triads are balanced or unbalanced, and counting components. The tasks are structured to be easy for a human who understands the underlying graph theoretic concepts, and can all be solved by counting the appropriate elements in graphs. We find that while GPT-4 consistently outperforms LLaVa, both models struggle with every visual network analysis task we propose. We publicly release the first benchmark for the evaluation of VLMs on foundational VNA tasks.