ActionEngine: From Reactive to Programmatic GUI Agents via State Machine Memory
On the map: LLM reasoning → Multimodal reasoning
also in 2 more
Abstract (arXiv)
Many web agents operate through a reactive execution loop: they observe the current interface, reason about the next action, execute it, and repeat. This design incurs latency and cost that grow with the number of actions, while requiring agents to repeatedly rediscover how the same web application works. We present ActionEngine, a novel architecture that replaces step-by-step reasoning with programmatic execution using reusable knowledge of the application. A Crawling Agent explores the application offline and constructs an updatable state-machine memory that represents its GUI states, the operations available in each state, and the transitions between states. Unlike trajectory memory, this representation stores how the application works rather than solutions to individual tasks. At runtime, an Execution Agent uses this memory to synthesize a complete executable program in a single planning step, which is then executed deterministically without further planning calls. When the interface changes or the memory is incomplete, a reactive fallback repairs the failed action and updates the memory for future tasks. On 655 tasks across four WebArena domains, ActionEngine achieves a 91.2% success rate, outperforming the strongest reactive baseline, Claude Code, by 8.5 percentage points while reducing average task latency by 3.2x and cost by 8x.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 17 of its 23 references that are in the map. First posted on arXiv 24 Feb 2026.
Builds on
17 of 23 references.
References · 17
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
- 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
- 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
- 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
- 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 553
- 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 350
- 2023Synapse: Trajectory-as-Exemplar Prompting with Memory for Computer Controlcited by 180
- 2023AutoDroid: LLM-powered Task Automation in Androidcited by 178
- 2024Is Your LLM Secretly a World Model of the Internet? Model-Based Planning for Web Agentscited by 144
- 2024Web Agents with World Models: Learning and Leveraging Environment Dynamics in Web Navigationcited by 116
- 2024AgentOccam: A Simple Yet Strong Baseline for LLM-Based Web Agentscited by 110
- 2024ScreenAgent: A Vision Language Model-driven Computer Control Agentcited by 101
5 more
- 2024GUI Agents: A Surveycited by 101
- 2025AgentRewardBench: Evaluating Automatic Evaluations of Web Agent Trajectoriescited by 99
- 2025Establishing Best Practices for Building Rigorous Agentic Benchmarkscited by 94
- 2025Symbiotic Cooperation for Web Agents: Harnessing Complementary Strengths of Large and Small LLMscited by 24
- 2025PG-Agent: An Agent Powered by Page Graphcited by 6
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2025Chain-of-Memory: Enhancing GUI Agents for Cross-Application Navigation
- 2025AppAgentX: Evolving GUI Agents as Proficient Smartphone Users
- 2026Efficient GUI Agents: A Systems Survey of Observation, Memory, Action, and Runtime Optimization
- 2026MAGNET: Towards Adaptive GUI Agents with Memory-Driven Knowledge Evolution
- 2025MMBench-GUI: Hierarchical Multi-Platform Evaluation Framework for GUI Agents
- 2025MGA: Memory-Driven GUI Agent for Observation-Centric Interaction
- 2025Breaking the Data Barrier -- Building GUI Agents Through Task Generalization
- 2026Benchmarking and Improving GUI Agents in High-Dynamic Environments