# InSight: A Benchmark for Agentic Claim Verification in Interactive Visualizations

arXiv: https://arxiv.org/abs/2609.01383 · HTML: https://mlsanity.com/paper/2609.01383/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 21 of its 34 references that are in the map.
- First posted on arXiv 1 Sep 2026.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)
- LLM agents and tool use / [Multimodal and embodied agents](https://mlsanity.com/agents/multimodal-and-embodied-agents/)

## Builds on (most cited first, 21 of 21)

- [ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning](https://mlsanity.com/paper/2203.10244/) (2022)
- [FEVER: a large-scale dataset for Fact Extraction and VERification](https://mlsanity.com/paper/1803.05355/) (2018)
- [DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing](https://mlsanity.com/paper/2111.09543/) (2021)
- [OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments](https://mlsanity.com/paper/2404.07972/) (2024)
- [Fact or Fiction: Verifying Scientific Claims](https://mlsanity.com/paper/2004.14974/) (2020)
- [On Evaluation of Embodied Navigation Agents](https://mlsanity.com/paper/1807.06757/) (2018)
- [DVQA: Understanding Data Visualizations via Question Answering](https://mlsanity.com/paper/1801.08163/) (2018)
- [PlotQA: Reasoning over Scientific Plots](https://mlsanity.com/paper/1909.00997/) (2019)
- [FigureQA: An Annotated Figure Dataset for Visual Reasoning](https://mlsanity.com/paper/1710.07300/) (2017)
- [Large Language Models are Better Reasoners with Self-Verification](https://mlsanity.com/paper/2212.09561/) (2022)
- [FEVEROUS: Fact Extraction and VERification Over Unstructured and Structured information](https://mlsanity.com/paper/2106.05707/) (2021)
- [The BrowserGym Ecosystem for Web Agent Research](https://mlsanity.com/paper/2412.05467/) (2024)
- [Multimodal Web Navigation with Instruction-Finetuned Foundation Models](https://mlsanity.com/paper/2305.11854/) (2023)
- [VisText: A Benchmark for Semantically Rich Chart Captioning](https://mlsanity.com/paper/2307.05356/) (2023)
- [NewsCLIPpings: Automatic Generation of Out-of-Context Multimodal Media](https://mlsanity.com/paper/2104.05893/) (2021)
- [Accessible Visualization via Natural Language Descriptions: A Four-Level Model of Semantic Content](https://mlsanity.com/paper/2110.04406/) (2021)
- [Visual Test-time Scaling for GUI Agent Grounding](https://mlsanity.com/paper/2505.00684/) (2025)
- [ChartCheck: Explainable Fact-Checking over Real-World Chart Images](https://mlsanity.com/paper/2311.07453/) (2023)
- [How Good is ChatGPT in Giving Advice on Your Visualization Design?](https://mlsanity.com/paper/2310.09617/) (2023)
- [AVerImaTeC: A Dataset for Automatic Verification of Image-Text Claims with Evidence from the Web](https://mlsanity.com/paper/2505.17978/) (2025)
- [ChartAgent: A Multimodal Agent for Visually Grounded Reasoning in Complex Chart Question Answering](https://mlsanity.com/paper/2510.04514/) (2025)

## Abstract (arXiv)

Vision Language Models have demonstrated remarkable proficiency in interpreting static visual artifacts, but modern data analysis is inherently dynamic, requiring the active interrogation of interactive environments. Existing benchmarks are predominantly constrained to static imagery and one-shot question answering and fail to capture the epistemic demands of this domain, where evidence is frequently occluded, distributed across linked views, or conditionally revealed through user agency. In this paper, we introduce InSight, a benchmark for agentic claim verification over interactive visualizations. The dataset consists of 21,349 claims derived from human-authored analytical narratives and grounded in fully interactive web-based environments. Agents must navigate these environments to determine whether a natural language claim is supported, refuted or not verifiable given the available evidence. Unlike traditional evaluations, InSight treats interaction traces as intrinsic proxies for reasoning, enabling a rigorous audit of how models seek and synthesize visual evidence. We evaluate state-of-the-art models, revealing that interactive verification remains a non-trivial challenge. We release InSight at https://github.com/maevehutch/insight.