# How Good (Or Bad) Are LLMs at Detecting Misleading Visualizations? arXiv: https://arxiv.org/abs/2407.17291 ยท HTML: https://mlsanity.com/paper/2407.17291/ ## Key facts - Cited by 27 papers in the map (data through 14 Oct 2026). - Placed in Chart understanding and anomaly detection, one of 16 directions in Multimodal LLMs / vision-language models. - Builds on 19 of its 38 references that are in the map. - First posted on arXiv 24 Jul 2024. ## Placed in - Multimodal LLMs / vision-language models / [Chart understanding and anomaly detection](https://mlsanity.com/mllm/chart-understanding-and-anomaly-detection/) ## Builds on (most cited first, 19 of 19) - [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023) - [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022) - [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022) - [ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning](https://mlsanity.com/paper/2203.10244/) (2022) - [The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)](https://mlsanity.com/paper/2309.17421/) (2023) - [Learning To Retrieve Prompts for In-Context Learning](https://mlsanity.com/paper/2112.08633/) (2021) - [Mixture-of-Agents Enhances Large Language Model Capabilities](https://mlsanity.com/paper/2406.04692/) (2024) - [PlotQA: Reasoning over Scientific Plots](https://mlsanity.com/paper/1909.00997/) (2019) - [FigureQA: An Annotated Figure Dataset for Visual Reasoning](https://mlsanity.com/paper/1710.07300/) (2017) - [UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoning](https://mlsanity.com/paper/2305.14761/) (2023) - [MatCha: Enhancing Visual Language Pretraining with Math Reasoning and Chart Derendering](https://mlsanity.com/paper/2212.09662/) (2022) - [Chart Question Answering: State of the Art and Future Directions](https://mlsanity.com/paper/2205.03966/) (2022) - [VizLinter: A Linter and Fixer Framework for Data Visualization](https://mlsanity.com/paper/2108.10299/) (2021) - [Misinformed by Visualization: What Do We Learn From Misinformative Visualizations?](https://mlsanity.com/paper/2204.09548/) (2022) - [Do LLMs Work on Charts? Designing Few-Shot Prompts for Chart Question Answering and Summarization](https://mlsanity.com/paper/2312.10610/) (2023) - [Viral Visualizations: How Coronavirus Skeptics Use Orthodox Data Practices to Promote Unorthodox Science Online](https://mlsanity.com/paper/2101.07993/) (2021) - [Why Change My Design: Explaining Poorly Constructed Visualization Designs with Explorable Explanations](https://mlsanity.com/paper/2309.01445/) (2023) - [Truncating the Y-Axis: Threat or Menace?](https://mlsanity.com/paper/1907.02035/) (2019) - [GeoLinter: A Linting Framework for Choropleth Maps](https://mlsanity.com/paper/2310.13707/) (2023) ## Built on by (most cited first, 27 of 27) - [Benchmarking Visual Language Models on Standardized Visualization Literacy Tests](https://mlsanity.com/paper/2503.16632/) (2025) - [Unmasking Deceptive Visuals: Benchmarking Multimodal Large Language Models on Misleading Chart Question Answering](https://mlsanity.com/paper/2503.18172/) (2025) - [Probing the Visualization Literacy of Vision Language Models: the Good, the Bad, and the Ugly](https://mlsanity.com/paper/2504.05445/) (2025) - [Protecting multimodal large language models against misleading visualizations](https://mlsanity.com/paper/2502.20503/) (2025) - [See or Recall: A Sanity Check for the Role of Vision in Solving Visualization Question Answer Tasks with Multimodal LLMs](https://mlsanity.com/paper/2504.09809/) (2025) - [The Perils of Chart Deception: How Misleading Visualizations Affect Vision-Language Models](https://mlsanity.com/paper/2508.09716/) (2025) - [Is this chart lying to me? Automating the detection of misleading visualizations](https://mlsanity.com/paper/2508.21675/) (2025) - [Evaluating LLMs for Visualization Generation and Understanding](https://mlsanity.com/paper/2507.22890/) (2025) - [Understanding Why ChatGPT Outperforms Humans in Visualization Design Advice](https://mlsanity.com/paper/2508.01547/) (2025) - [VizDefender: Unmasking Visualization Tampering through Proactive Localization and Intent Inference](https://mlsanity.com/paper/2512.18853/) (2025) - [LLM-Assisted Visual Analytics: Opportunities and Challenges](https://mlsanity.com/paper/2409.02691/) (2024) - [When AI reviews science: Can we trust the referee?](https://mlsanity.com/paper/2604.23593/) (2026) - [Do Language Model Agents Align with Humans in Rating Visualizations? An Empirical Study](https://mlsanity.com/paper/2505.06702/) (2025) - [Navigating the Mirage: A Dual-Path Agentic Framework for Robust Misleading Chart Question Answering](https://mlsanity.com/paper/2603.28583/) (2026) - [Automated Visualization Makeovers with LLMs](https://mlsanity.com/paper/2508.05637/) (2025) - [Benchmark It Yourself (BIY): Preparing a Dataset and Benchmarking AI Models for Scatterplot-Related Tasks](https://mlsanity.com/paper/2510.06071/) (2025) - [ChartAttack: Testing the Vulnerability of LLMs to Malicious Prompting in Chart Generation](https://mlsanity.com/paper/2601.12983/) (2026) - [Do Large Language Models Understand Data Visualization Principles?](https://mlsanity.com/paper/2602.20084/) (2026) - [Do Large Language Models Understand Data Visualization Rules?](https://mlsanity.com/paper/2602.20137/) (2026) - [De-rendering, Reasoning, and Repairing Charts with Vision-Language Models](https://mlsanity.com/paper/2602.20291/) (2026) - [When Visuals Aren't the Problem: Evaluating Vision-Language Models on Misleading Data Visualizations](https://mlsanity.com/paper/2603.22368/) (2026) - [IPO-Mine: A Toolkit and Dataset for Section-Structured Analysis of Long, Multimodal IPO Documents](https://mlsanity.com/paper/2605.28714/) (2026) - [Disentangling Visual and Factual Correctness in LVLMs' Visualization Literacy](https://mlsanity.com/paper/2606.03142/) (2026) - [EvidenceLens: A Claim-Evidence Matrix for Auditing Financial Question Answering](https://mlsanity.com/paper/2606.23724/) (2026) - [Attention-Guided Saliency Maps for Interpreting Visualization Literacy in VLMs](https://mlsanity.com/paper/2607.16105/) (2026) - [Chart Deception in Vision-Language Models: From Vulnerability to Mitigation](https://mlsanity.com/paper/2607.22600/) (2026) - [Exploring Agentic Approaches for Data Issue Detection and Repair in AI-Assisted Visualization](https://mlsanity.com/paper/2608.21602/) (2026) ## Abstract (arXiv) In this study, we address the growing issue of misleading charts, a prevalent problem that undermines the integrity of information dissemination. Misleading charts can distort the viewer's perception of data, leading to misinterpretations and decisions based on false information. The development of effective automatic detection methods for misleading charts is an urgent field of research. The recent advancement of multimodal Large Language Models (LLMs) has introduced a promising direction for addressing this challenge. We explored the capabilities of these models in analyzing complex charts and assessing the impact of different prompting strategies on the models' analyses. We utilized a dataset of misleading charts collected from the internet by prior research and crafted nine distinct prompts, ranging from simple to complex, to test the ability of four different multimodal LLMs in detecting over 21 different chart issues. Through three experiments--from initial exploration to detailed analysis--we progressively gained insights into how to effectively prompt LLMs to identify misleading charts and developed strategies to address the scalability challenges encountered as we expanded our detection range from the initial five issues to 21 issues in the final experiment. Our findings reveal that multimodal LLMs possess a strong capability for chart comprehension and critical thinking in data interpretation. There is significant potential in employing multimodal LLMs to counter misleading information by supporting critical thinking and enhancing visualization literacy. This study demonstrates the applicability of LLMs in addressing the pressing concern of misleading charts.