mlsanityData to 14 Oct 2026

When Charts Deceive Web Agents: A Paired Benchmark for Misleading Visualization-Grounded Tasks

arXiv 2610.12823 ↗First posted 9 Oct 2026cs.CLcited by 0

On the map: LLM agents and tool use → Web, search, and research agents

Abstract (arXiv)

Web agents increasingly browse, interpret, and act on web content on behalf of users. In many realistic workflows, however, the visual evidence they consume is not neutral: charts and dashboards can be misleading, shaping judgments without issuing any explicit malicious instruction. Unlike chart question answering, Web Agents treat charts as intermediate evidence whose misinterpretation can propagate into concrete downstream actions --- a reliability risk that remains underexplored. We introduce MisVisAgentBench, the first paired executable benchmark for evaluating how misleading visualizations affect Web Agent execution. It comprises 280 task instances in matched clean--misleading pairs that differ only in chart evidence. We further provide a diagnostic framework that isolates misleading-induced failures from baseline task difficulty (Zhu et al., 2025) and captures execution-level effects such as extra browsing steps and decision hesitation. Across thirteen vision-language Web Agents, replacing clean charts with misleading counterparts reduces the average task success rate by more than 34 percentage points. Moreover, these failures predominantly follow the targeted misleading action branch, suggesting that deceptive charts systematically redirect rather than randomly disrupt agent decisions. These results reveal that visualization robustness is a critical yet understudied dimension of Web Agent safety and reliability.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use. Builds on 31 of its 54 references that are in the map. First posted on arXiv 9 Oct 2026.

Builds on

31 of 54 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023WebArena: A Realistic Web Environment for Building Autonomous Agents25 Jul
  2. 2023Mind2Web: Towards a Generalist Agent for the Web9 Jun

Other references · 29

  1. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  2. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,195
  3. 2025BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agentscited by 647
  4. 2018DVQA: Understanding Data Visualizations via Question Answeringcited by 340
  5. 2019PlotQA: Reasoning over Scientific Plotscited by 321
  6. 2023UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoningcited by 160
  7. 2025WASP: Benchmarking Web Agent Security Against Prompt Injection Attackscited by 157
  8. 2024ST-WebAgentBench: A Benchmark for Evaluating Safety and Trustworthiness in Web Agentscited by 133
  9. 2023ChartBench: A Benchmark for Complex Visual Reasoning in Chartscited by 120
  10. 2025ChartQAPro: A More Diverse and Challenging Benchmark for Chart Question Answeringcited by 108
  11. 2025SafeArena: Evaluating the Safety of Autonomous Web Agentscited by 102
  12. 2025The Obvious Invisible Threat: LLM-Powered GUI Agents' Vulnerability to Fine-Print Injectionscited by 59
17 more
  1. 2025The LLM Already Knows: Estimating LLM-Perceived Question Difficulty via Hidden Representationscited by 29
  2. 2019Dark Patterns at Scale: Findings from a Crawl of 11K Shopping Websitescited by 19
  3. 2025Investigating the Impact of Dark Patterns on LLM-Based Web Agentscited by 17
  4. 2025Unmasking Deceptive Visuals: Benchmarking Multimodal Large Language Models on Misleading Chart Question Answeringcited by 16
  5. 2026Youtu-VL: Unleashing Visual Potential via Unified Vision-Language Supervisioncited by 15
  6. 2025EncQA: Benchmarking Vision-Language Models on Visual Encodings for Chartscited by 12
  7. 2018Ethical Dimensions of Visualization Researchcited by 10
  8. 2025Prompt-R1: Collaborative Automatic Prompting Framework via End-to-end Reinforcement Learningcited by 10
  9. 2025Protecting multimodal large language models against misleading visualizationscited by 8
  10. 2025ChartMind: A Comprehensive Benchmark for Complex Real-world Multimodal Chart Question Answeringcited by 7
  11. 2025The Perils of Chart Deception: How Misleading Visualizations Affect Vision-Language Modelscited by 7
  12. 2025Is this chart lying to me? Automating the detection of misleading visualizationscited by 6
  13. 2025LexGenius: An Expert-Level Benchmark for Large Language Models in Legal General Intelligencecited by 4
  14. 2026TIEM: Temporal Integration of Hypergraph Evidence and Skill Memory for Event-Driven Financial Forecastingcited by 2
  15. 2026When Does Visual Generation Help Visual Understanding in Unified Multimodal Models?cited by 2
  16. 2026UVU: Improving Multimodal Understanding via Vision-Language Unified Autoregressive Paradigmcited by 1
  17. 2026VIVAS: Vitalizing Visual Perception in VLM Pre-training via Vision-language Unified Autoregressive Supervisioncited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026VizAnchor: Decoding Manipulation Intent from Tampering Visualizations via Dual-Anchor Reasoning
  2. 2026Do Web Agents Investigate Before They Decide?
  3. 2026When Web Agents Finish but Still Fail: Reproducible Triggers and Trace Diagnostics for Parallel Web Exploration
  4. 2026InSight: A Benchmark for Agentic Claim Verification in Interactive Visualizations
  5. 2026The Hard Part Comes After Search: Benchmarking Web Agents on Synthesizing, Organizing, and Displaying Knowledge
  6. 2026How do Visual Attributes Influence Web Agents? A Comprehensive Evaluation of User Interface Design Factors
  7. 2026DRNOISE: Benchmarking Deep Research Agents in Misleading Evidence Environments
  8. 2025DECEPTICON: How Dark Patterns Manipulate Web Agents