# Benchmarking and Improving GUI Agents in High-Dynamic Environments

arXiv: https://arxiv.org/abs/2604.25380 · HTML: https://mlsanity.com/paper/2604.25380/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 31 of its 46 references that are in the map.
- First posted on arXiv 28 Apr 2026.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)
- LLM agents and tool use / [GUI and mobile agents](https://mlsanity.com/agents/gui-and-mobile-agents/)

## Builds on (most cited first, 30 of 31)

- [Mind2Web: Towards a Generalist Agent for the Web](https://mlsanity.com/paper/2306.06070/) (2023)
- [CogAgent: A Visual Language Model for GUI Agents](https://mlsanity.com/paper/2312.08914/) (2023)
- [UI-TARS: Pioneering Automated GUI Interaction with Native Agents](https://mlsanity.com/paper/2501.12326/) (2025)
- [AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents](https://mlsanity.com/paper/2405.14573/) (2024)
- [Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents](https://mlsanity.com/paper/2410.05243/) (2024)
- [Aguvis: Unified Pure Vision Agents for Autonomous GUI Interaction](https://mlsanity.com/paper/2412.04454/) (2024)
- [GUI-R1 : A Generalist R1-Style Vision-Language Action Model For GUI Agents](https://mlsanity.com/paper/2504.10458/) (2025)
- [ShowUI: One Vision-Language-Action Model for GUI Visual Agent](https://mlsanity.com/paper/2411.17465/) (2024)
- [GUIOdyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devices](https://mlsanity.com/paper/2406.08451/) (2024)
- [Mobile-Agent-v3: Fundamental Agents for GUI Automation](https://mlsanity.com/paper/2508.15144/) (2025)
- [OpenCUA: Open Foundations for Computer-Use Agents](https://mlsanity.com/paper/2508.09123/) (2025)
- [Aria-UI: Visual Grounding for GUI Instructions](https://mlsanity.com/paper/2412.16256/) (2024)
- [Scaling Computer-Use Grounding via User Interface Decomposition and Synthesis](https://mlsanity.com/paper/2505.13227/) (2025)
- [GTA1: GUI Test-time Scaling Agent](https://mlsanity.com/paper/2507.05791/) (2025)
- [Enhancing Visual Grounding for GUI Agents via Self-Evolutionary Reinforcement Learning](https://mlsanity.com/paper/2505.12370/) (2025)
- [Mobile-Agent-v3.5: Multi-platform Fundamental GUI Agents](https://mlsanity.com/paper/2602.16855/) (2026)
- [EvoCUA: Evolving Computer Use Agents via Learning from Scalable Synthetic Experience](https://mlsanity.com/paper/2601.15876/) (2026)
- [Seed1.8 Model Card: Towards Generalized Real-World Agency](https://mlsanity.com/paper/2603.20633/) (2026)
- [TongUI: Internet-Scale Trajectories from Multimodal Web Tutorials for Generalized GUI Agents](https://mlsanity.com/paper/2504.12679/) (2025)
- [CoAct-1: Computer-using Multi-Agent System with Coding Actions](https://mlsanity.com/paper/2508.03923/) (2025)
- [UI-AGILE: Advancing GUI Agents with Effective Reinforcement Learning and Precise Inference-Time Grounding](https://mlsanity.com/paper/2507.22025/) (2025)
- [SpiritSight Agent: Advanced GUI Agent with One Look](https://mlsanity.com/paper/2503.03196/) (2025)
- [GUI-Robust: A Comprehensive Dataset for Testing GUI Agent Robustness in Real-World Anomalies](https://mlsanity.com/paper/2506.14477/) (2025)
- [Chain-of-Memory: Enhancing GUI Agents for Cross-Application Navigation](https://mlsanity.com/paper/2506.18158/) (2025)
- [GUI-ReWalk: Massive Data Generation for GUI Agent via Stochastic Exploration and Intent-Aware Reasoning](https://mlsanity.com/paper/2509.15738/) (2025)
- [Learning GUI Grounding with Spatial Reasoning from Visual Feedback](https://mlsanity.com/paper/2509.21552/) (2025)
- [MobileA3gent: Training Mobile GUI Agents Using Decentralized Self-Sourced Data from Diverse Users](https://mlsanity.com/paper/2502.02982/) (2025)
- [textscGUI-Spotlight: Adaptive Iterative Focus Refinement for Enhanced GUI Visual Grounding](https://mlsanity.com/paper/2510.04039/) (2025)
- [HATS: Hardness-Aware Trajectory Synthesis for GUI Agents](https://mlsanity.com/paper/2603.12138/) (2026)
- [D-GARA: A Dynamic Benchmarking Framework for GUI Agent Robustness in Real-World Anomalies](https://mlsanity.com/paper/2511.16590/) (2025)

## Built on by (most cited first, 2 of 2)

- [Why Are GUI Agents Correct but Late? Decode on the Decision-Time Critical Path, Tested with Pre-Compiled Policy Trees](https://mlsanity.com/paper/2607.28399/) (2026)
- [Learn from Move: the Next Step for GUI Agents](https://mlsanity.com/paper/2610.16720/) (2026)

## Abstract (arXiv)

Recent advancements in Graphical User Interface (GUI) agents have predominantly focused on training paradigms like supervised fine-tuning (SFT) and reinforcement learning (RL). However, the challenge of high-dynamic GUI environments remains largely underexplored. Existing agents typically rely on a single screenshot after each action for decision-making, leading to a partially observable (or even unobservable) Markov decision process, where the key GUI state including important information for actions is often inadequately captured. To systematically explore this challenge, we introduce DynamicGUIBench, a comprehensive online GUI benchmark spanning ten applications and diverse interaction scenarios characterized by important interface changes between actions. Furthermore, we present DynamicUI, an agent designed for dynamic interfaces, which takes screen-recording videos of the interaction process as input and consists of three components: a dynamic perceiver, a refinement strategy, and a reflection. Specifically, the dynamic perceiver clusters frames of the GUI video, generates captions for the centroids, and iteratively selects the most informative frames as the salient dynamic context. Considering that there may be inconsistencies and noise between the selected frames and the textual context of the agent, the refinement strategy employs an action-conditioned filtering to refine thoughts to mitigate thought-action inconsistency and redundancy. Based on the refined agent trajectories, the reflection module provides effective and accurate guidance for further actions. Experiments on DynamicGUIBench demonstrate that DynamicUI significantly improves the performance in dynamic GUI environments, while maintaining competitive performance on other public benchmarks.