mlsanityData to 14 Oct 2026

Losing the Plot: How VLM responses degrade on imperfect charts

arXiv 2509.18425 ↗First posted 22 Sep 2025cs.CVcited by 5

On the map: Multimodal LLMs / vision-language models → Chart understanding and anomaly detection

Abstract (arXiv)

Vision language models (VLMs) show strong results on chart understanding, yet existing benchmarks assume clean figures and fact based queries. Real world charts often contain distortions and demand reasoning beyond simple matching. We evaluate ChatGPT 4o, Claude Sonnet 4, and Gemini 2.5 Pro, finding sharp performance drops under corruption or occlusion, with hallucinations such as value fabrication, trend misinterpretation, and entity confusion becoming more frequent. Models remain overconfident in degraded settings, generating plausible but unsupported explanations. To address this gap, we introduce CHART NOISe(Chart Hallucinations, Answers, and Reasoning Testing on Noisy and Occluded Input Selections), a dataset combining chart corruptions, occlusions, and exam style multiple choice questions inspired by Korea's CSAT English section. A key innovation is prompt reverse inconsistency, where models contradict themselves when asked to confirm versus deny the same statement. Our contributions are threefold: (1) benchmarking state of the art VLMs, exposing systematic vulnerabilities in chart reasoning; (2) releasing CHART NOISe, the first dataset unifying corruption, occlusion, and reverse inconsistency; and (3) proposing baseline mitigation strategies such as quality filtering and occlusion detection. Together, these efforts establish a rigorous testbed for advancing robustness and reliability in chart understanding.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in Chart understanding and anomaly detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 21 of its 27 references that are in the map. First posted on arXiv 22 Sep 2025.

Builds on

21 of 27 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Socratic Chart: Cooperating Multiple Agents for Robust SVG Chart Understanding14 Apr
  2. 2024CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMs26 Jun
  3. 2024ChartAssisstant: A Universal Chart Multimodal Language Model via Chart-to-Table Pre-training and Multitask Instruction Tuning4 Jan
  4. 2023UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoning24 May

Other references · 17

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,877
  3. 2015Learning Deep Features for Discriminative Localizationcited by 2,436
  4. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  5. 2024Hallucination of Multimodal Large Language Models: A Surveycited by 388
  6. 2024A Survey on Hallucination in Large Vision-Language Modelscited by 357
  7. 2019PlotQA: Reasoning over Scientific Plotscited by 321
  8. 2022DePlot: One-shot visual language reasoning by plot-to-table translationcited by 160
  9. 2025ChartQAPro: A More Diverse and Challenging Benchmark for Chart Question Answeringcited by 108
  10. 2023ARNIQA: Learning Distortion Manifold for Image Quality Assessmentcited by 89
  11. 2024Reducing Hallucinations in Vision-Language Models via Latent Space Steeringcited by 59
  12. 2024Mitigating Object Hallucination in Large Vision-Language Models via Image-Grounded Guidancecited by 46
5 more
  1. 2025CAPTURe: Evaluating Spatial Reasoning in Vision Language Models via Occluded Object Countingcited by 38
  2. 2024B-AVIBench: Towards Evaluating the Robustness of Large Vision-Language Model on Black-box Adversarial Visual-Instructionscited by 26
  3. 2025SECOND: Mitigating Perceptual Hallucination in Vision-Language Models via Selective and Contrastive Decodingcited by 19
  4. 2025Prompt-Reverse Inconsistency: LLM Self-Inconsistency Beyond Generative Randomness and Prompt Paraphrasingcited by 14
  5. 2025Disharmony: Forensics using Reverse Lighting Harmonizationcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026Failing to See or Failing to Know? Attributing Errors in Vision-Language Modelscited by 1
  2. 2026Toward a Machine Bertin: Why Visualization Needs Design Principles for Machine Cognitioncited by 0
  3. 2026When Relations Break: Analyzing Relation Hallucination in Vision-Language Model Under Rotation and Noisecited by 0
  4. 2026When Does Consensus Mean Correctness? Measuring the Agreement-Accuracy Coupling with Semantics-Preserving Re-Renderingcited by 0
  5. 2026Consistency Has a Computable Blind Spot: A Commutation Theory of Label-Free Reliability for Vision-Language Figure Readingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Chartographer: Counterfactual Chart Generation for Evaluating Vision-Language Models
  2. 2025Unmasking Deceptive Visuals: Benchmarking Multimodal Large Language Models on Misleading Chart Question Answering
  3. 2024Unraveling the Truth: Do VLMs really Understand Charts? A Deep Dive into Consistency and Robustness
  4. 2025Judging the Judges: Can Large Vision-Language Models Fairly Evaluate Chart Comprehension and Reasoning?
  5. 2025FinChart-Bench: Benchmarking Financial Chart Comprehension in Vision-Language Models
  6. 2025InterChart: Benchmarking Visual Reasoning Across Decomposed and Distributed Chart Information
  7. 2026ChartVerse: Scaling Chart Reasoning via Reliable Programmatic Synthesis from Scratch
  8. 2025ChartMuseum: Testing Visual Reasoning Capabilities of Large Vision-Language Models