DashAct: A Progressive Diagnostic Benchmark for GUI Agents in Interactive Dashboard Analysis
On the map: LLM agents and tool use → GUI and mobile agents
Abstract (arXiv)
Interactive dashboards require users to reveal and connect evidence across stateful interactions. Although graphical user interface (GUI) agents could automate this process, existing dashboard benchmarks primarily report final answers or task success. They provide limited insight into whether failures arise from maintaining the analytical process, selecting actions, or grounding visual targets. We introduce DashAct, to our knowledge the first benchmark to diagnose these failures at a fine-grained level within the same dashboard task. DashAct contains 357 human-verified interaction trajectories with milestone dependencies and hierarchical target annotations. Its progressive diagnostic cascade evaluates end-to-end execution, restores verified context for next-action prediction, and provides target semantics and a local view for visual grounding. By progressively restoring the conditions for success, DashAct measures the minimum support an agent needs to recover rather than scoring isolated skills. Experiments show that current models struggle even as support is added. The cascade outcomes reveal bottlenecks hidden by end-to-end scores and provide actionable guidance for improving GUI agents.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use. Builds on 21 of its 30 references that are in the map. First posted on arXiv 26 Sep 2026.
Builds on
21 of 30 references.
Closest ancestors · 6
Same direction, newest first
- 2026Mobile-Agent-v3.5: Multi-platform Fundamental GUI Agents15 Feb
- 2025OpenCUA: Open Foundations for Computer-Use Agents12 Aug
- 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agents21 Jan
- 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents7 Oct
- 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments11 Apr
- 2023CogAgent: A Visual Language Model for GUI Agents14 Dec
Other references · 15
- 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
- 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 468
- 2024WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?cited by 363
- 2018DVQA: Understanding Data Visualizations via Question Answeringcited by 340
- 2019PlotQA: Reasoning over Scientific Plotscited by 321
- 2017FigureQA: An Annotated Figure Dataset for Visual Reasoningcited by 319
- 2024ShowUI: One Vision-Language-Action Model for GUI Visual Agentcited by 232
- 2024WebLINX: Real-World Website Navigation with Multi-Turn Dialoguecited by 180
- 2023VisText: A Benchmark for Semantically Rich Chart Captioningcited by 112
- 2024VideoGUI: A Benchmark for GUI Automation from Instructional Videoscited by 35
- 2025ProactiveVA: Proactive Visual Analytics with LLM-Based UI Agentcited by 14
- 2025RefChartQA: Grounding Visual Answer on Chart Images through Instruction Tuningcited by 11
3 more
- 2026OpAgent: Operator Agent for Web Navigationcited by 8
- 2025ChartPoint: Guiding MLLMs with Grounding Reflection for Chart Reasoningcited by 6
- 2026POINTS-GUI-G: GUI-Grounding Journeycited by 4
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026DashArena: Benchmarking LLMs on Interactive Analytic Dashboard Generation
- 2026Covering Human Action Space for Computer Use: Data Synthesis and Benchmark
- 2025GUI-360^circ: A Comprehensive Dataset and Benchmark for Computer-Using Agents
- 2025GUI-Robust: A Comprehensive Dataset for Testing GUI Agent Robustness in Real-World Anomalies
- 2026GUIDE: A Benchmark for Understanding and Assisting Users in Open-Ended GUI Tasks
- 2026GUIDE: Interpretable GUI Agent Evaluation via Hierarchical Diagnosis
- 2025ScreenLLM: Stateful Screen Schema for Efficient Action Understanding and Prediction
- 2026Dashboard2Code: Evaluating Multimodal Models on Reconstructing Interactive Dashboards