mlsanityData to 7 Oct 2026

CHARTOM: A Visual Theory-of-Mind Benchmark for LLMs on Misleading Charts

arXiv 2408.14419 ↗First posted 26 Aug 2024cs.AI cs.CL cs.CVcited by 5

On the map: Multimodal LLMs / vision-language models → Chart understanding and anomaly detection

Abstract (arXiv)

We introduce CHARTOM, a visual theory-of-mind benchmark designed to evaluate multimodal large language models' capability to understand and reason about misleading data visualizations though charts. CHARTOM consists of carefully designed charts and associated questions that require a language model to not only correctly comprehend the factual content in the chart (the FACT question) but also judge whether the chart will be misleading to a human readers (the MIND question), a dual capability with significant societal benefits. We detail the construction of our benchmark including its calibration on human performance and estimation of MIND ground truth called the Human Misleadingness Index. We evaluated several leading LLMs -- including GPT, Claude, Gemini, Qwen, Llama, and Llava series models -- on the CHARTOM dataset and found that it was challenging to all models both on FACT and MIND questions. This highlights the limitations of current LLMs and presents significant opportunity for future LLMs to improve on understanding misleading charts.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Chart understanding and anomaly detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 11 of its 43 references that are in the map. First posted on arXiv 26 Aug 2024.

Builds on

11 of 43 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Are Large Vision Language Models up to the Challenge of Chart Comprehension and Reasoning? An Extensive Investigation into the Capabilities and Limitations of LVLMs1 Jun
  2. 2023UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoning24 May

Other references · 9

  1. 2023Large Language Models Fail on Trivial Alterations to Theory-of-Mind Taskscited by 288
  2. 2022Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMscited by 220
  3. 2023Understanding Social Reasoning in Language Models with Language Modelscited by 166
  4. 2023FANToM: A Benchmark for Stress-testing Machine Theory of Mind in Interactionscited by 162
  5. 2023How FaR Are Large Language Models From Agents with Theory-of-Mind?cited by 75
  6. 2024ToMBench: Benchmarking Theory of Mind in Large Language Modelscited by 70
  7. 2024From Pixels to Insights: A Survey on Automatic Chart Understanding in the Era of Large Foundation Modelscited by 52
  8. 2022Misinformed by Visualization: What Do We Learn From Misinformative Visualizations?cited by 19
  9. 2023Do LLMs Work on Charts? Designing Few-Shot Prompts for Chart Question Answering and Summarizationcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 3.

5 papers.

  1. 2025Protecting multimodal large language models against misleading visualizationscited by 6
  2. 2025Modeling the Mental World for Embodied AI: A Comprehensive Reviewcited by 1
  3. 2026Navigating the Mirage: A Dual-Path Agentic Framework for Robust Misleading Chart Question Answeringcited by 1
  4. 2026MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Interventioncited by 0
  5. 2026Beyond What Meets the Eye: Unveiling Situational Illusions for Multimodal Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Unmasking Deceptive Visuals: Benchmarking Multimodal Large Language Models on Misleading Chart Question Answering
  2. 2024How Good (Or Bad) Are LLMs at Detecting Misleading Visualizations?
  3. 2026When Visuals Aren't the Problem: Evaluating Vision-Language Models on Misleading Data Visualizations
  4. 2026Chart Deception in Vision-Language Models: From Vulnerability to Mitigation
  5. 2025ChartMuseum: Testing Visual Reasoning Capabilities of Large Vision-Language Models
  6. 2024CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMs
  7. 2025Losing the Plot: How VLM responses degrade on imperfect charts
  8. 2026ChartFI: Benchmarking Faithfulness and Insightfulness of Chart Descriptions from Multimodal Large Language Models