# Can GPT-4 Models Detect Misleading Visualizations?

arXiv: https://arxiv.org/abs/2408.12617 · HTML: https://mlsanity.com/paper/2408.12617/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 39 references that are in the map.
- First posted on arXiv 8 Aug 2024.

## Builds on (most cited first, 11 of 11)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [The spread of low-credibility content by social bots](https://mlsanity.com/paper/1707.07592/) (2017)
- [Do LVLMs Understand Charts? Analyzing and Correcting Factual Errors in Chart Captioning](https://mlsanity.com/paper/2312.10160/) (2023)
- [How Good is ChatGPT in Giving Advice on Your Visualization Design?](https://mlsanity.com/paper/2310.09617/) (2023)
- [Misinformed by Visualization: What Do We Learn From Misinformative Visualizations?](https://mlsanity.com/paper/2204.09548/) (2022)
- [Are LLMs ready for Visualization?](https://mlsanity.com/paper/2403.06158/) (2024)
- [Beneath the Surface: Unveiling Harmful Memes with Multimodal Reasoning Distilled from Large Language Models](https://mlsanity.com/paper/2312.05434/) (2023)
- [Ethical Dimensions of Visualization Research](https://mlsanity.com/paper/1811.07271/) (2018)
- [Do LLMs Work on Charts? Designing Few-Shot Prompts for Chart Question Answering and Summarization](https://mlsanity.com/paper/2312.10610/) (2023)
- [Comparing GPT-4 and Open-Source Language Models in Misinformation Mitigation](https://mlsanity.com/paper/2401.06920/) (2024)

## Built on by (most cited first, 10 of 10)

- [EncQA: Benchmarking Vision-Language Models on Visual Encodings for Charts](https://mlsanity.com/paper/2508.04650/) (2025)
- [Protecting multimodal large language models against misleading visualizations](https://mlsanity.com/paper/2502.20503/) (2025)
- [See or Recall: A Sanity Check for the Role of Vision in Solving Visualization Question Answer Tasks with Multimodal LLMs](https://mlsanity.com/paper/2504.09809/) (2025)
- [The Perils of Chart Deception: How Misleading Visualizations Affect Vision-Language Models](https://mlsanity.com/paper/2508.09716/) (2025)
- [Understanding Why ChatGPT Outperforms Humans in Visualization Design Advice](https://mlsanity.com/paper/2508.01547/) (2025)
- [Large Language Model-Informed Feature Discovery Improves Prediction and Interpretation of Credibility Perceptions of Visual Content](https://mlsanity.com/paper/2504.10878/) (2025)
- [The Role of Visualization in LLM-Assisted Knowledge Graph Systems: Effects on User Trust, Exploration, and Workflows](https://mlsanity.com/paper/2505.21512/) (2025)
- [Do Large Language Models Understand Data Visualization Principles?](https://mlsanity.com/paper/2602.20084/) (2026)
- [Do Large Language Models Understand Data Visualization Rules?](https://mlsanity.com/paper/2602.20137/) (2026)
- [Chart Deception in Vision-Language Models: From Vulnerability to Mitigation](https://mlsanity.com/paper/2607.22600/) (2026)

## Abstract (arXiv)

The proliferation of misleading visualizations online, particularly during critical events like public health crises and elections, poses a significant risk. This study investigates the capability of GPT-4 models (4V, 4o, and 4o mini) to detect misleading visualizations. Utilizing a dataset of tweet-visualization pairs containing various visual misleaders, we test these models under four experimental conditions with different levels of guidance. We show that GPT-4 models can detect misleading visualizations with moderate accuracy without prior training (naive zero-shot) and that performance notably improves when provided with definitions of misleaders (guided zero-shot). However, a single prompt engineering technique does not yield the best results for all misleader types. Specifically, providing the models with misleader definitions and examples (guided few-shot) proves more effective for reasoning misleaders, while guided zero-shot performs better for design misleaders. This study underscores the feasibility of using large vision-language models to detect visual misinformation and the importance of prompt engineering for optimized detection accuracy.