Beyond Pixels: Introspective and Interactive Grounding for Visualization Agents
On the map: Multimodal LLMs / vision-language models → Chart understanding and anomaly detection
also in 1 more
Abstract (arXiv)
Vision-Language Models (VLMs) frequently misread values, hallucinate details, and confuse overlapping elements in charts. Current approaches rely solely on pixel interpretation, creating a Pixel-Only Bottleneck: agents treat interactive charts as static images, losing access to the structured specification that encodes exact values. We introduce Introspective and Interactive Visual Grounding (IVG), a framework that combines (1) spec-grounded introspection, which queries the underlying specification for deterministic evidence, with (2) view-grounded interaction, which manipulates the view to resolve visual ambiguity. To enable evaluation without VLM bias, we present iPlotBench, a benchmark of 500 interactive Plotly figures with 6,706 binary questions and ground-truth specifications. Experiments show that introspection improves data reconstruction fidelity, while the combination with interaction achieves the highest QA accuracy (0.81), with +6.7 % gains on overlapping geometries. We further demonstrate IVG in deployed agents that explore data autonomously and collaborate with human users in real time.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Chart understanding and anomaly detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 17 of its 30 references that are in the map. First posted on arXiv 22 Apr 2026.
Builds on
17 of 30 references.
Closest ancestors · 1
Same direction, newest first
Other references · 16
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
- 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
- 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,628
- 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
- 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agentscited by 956
- 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 671
- 2024Agentless: Demystifying LLM-based Software Engineering Agentscited by 400
- 2018DVQA: Understanding Data Visualizations via Question Answeringcited by 338
- 2019PlotQA: Reasoning over Scientific Plotscited by 317
- 2017FigureQA: An Annotated Figure Dataset for Visual Reasoningcited by 314
- 2024CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challengescited by 278
- 2024Data Interpreter: An LLM Agent For Data Sciencecited by 222
4 more
- 2025RepoAudit: An Autonomous LLM-Agent for Repository-Level Code Auditingcited by 47
- 2025METAL: A Multi-Agent Framework for Chart Generation with Test-Time Scalingcited by 33
- 2025Automated Visualization Code Synthesis via Multi-Path Reasoning and Feedback-Driven Optimizationcited by 6
- 2023Draco 2: An Extensible Platform to Model Visualization Designcited by 5
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Connecting the Dots: Training-Free Visual Grounding via Agentic Reasoning
- 2025ChartAB: A Benchmark for Chart Grounding & Dense Alignment
- 2026When Seeing Is Not Enough: Benchmarking Interactive Visual Grounding in LVLMs
- 2026ChartREG++: Towards Benchmarking and Improving Chart Referring Expression Grounding under Diverse referring clues and Multi-Target Referring
- 2026ChartProbe: A Diagnostic Study on Visual Reasoning through Perception, Grounding, and Simple Reasoning
- 2025Do Vision-Language Models See Visualizations Like Humans? Alignment in Chart Categorization
- 2025Towards Understanding Visual Grounding in Visual Language Models
- 2025ChartAgent: A Multimodal Agent for Visually Grounded Reasoning in Complex Chart Question Answering