StructAgent: Harness Long-horizon Digital Agents with Unified Causal Structure
On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models
also in 1 more
Abstract (arXiv)
Recent advances in large language models (LLMs) and vision-language models (VLMs) have enabled increasingly capable digital agents for computer use. However, real-world tasks are often long-horizon and involve evolving contexts containing accumulated observations, intermediate edits, failed attempts, and partially completed executions. Existing agents typically operate over raw interaction history, making task progress difficult to interpret, verify, and recover, which ultimately limits reliable long-horizon execution. In this paper, we argue that addressing this challenge requires explicitly structuring both the agent's state and workflow around a unified causal representation of task progress. We present StructAgent, a state-centered framework that introduces a unified state for maintaining compact, verifiable task progress and a structured workflow that regulates progress through verifier-backed state transitions. Building on this design, StructAgent further enables explicit progress checkpointing, evidence-driven task completion, targeted failure recovery, and tool-supported execution, while ensuring that all progress updates remain grounded in verification. Extensive experiments demonstrate that StructAgent consistently improves a wide range of LLM and VLM backbones on long-horizon computer-use tasks. On OSWorld-Verified, it improves Qwen3.5-9B from 27.0% to 46.9% success rate and Qwen3.5-27B from 31.6% to 62.2%, while achieving a new open-source state of the art of 78.9% with MiniMax-M3. Moreover, the same framework generalizes beyond desktop environments to Minecraft, demonstrating the generality of our design.
Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 51 of its 54 references that are in the map. First posted on arXiv 13 Jul 2026.
Builds on
51 of 54 references.
Closest ancestors · 8
Same direction, newest first
- 2026Hybrid Self-evolving Structured Memory for GUI Agents11 Mar
- 2026OS-Symphony: A Holistic Framework for Robust and Generalist Computer-Using Agent12 Jan
- 2025EchoTrail-GUI: Building Actionable Memory for GUI Agents via Critic-Guided Self-Exploration22 Dec
- 2025MGA: Memory-Driven GUI Agent for Observation-Centric Interaction28 Oct
- 2025ScaleCUA: Scaling Open-Source Computer Use Agents with Cross-Platform Data18 Sep
- 2025OpenCUA: Open Foundations for Computer-Use Agents12 Aug
- 2025SEAgent: Self-Evolving Computer Use Agent with Autonomous Learning from Experience6 Aug
- 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agents30 Oct
Other references · 43
- 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
- 2025Qwen3-VL Technical Reportcited by 2,793
- 2023A Survey of Large Language Modelscited by 2,576
- 2017Billion-scale similarity search with GPUscited by 2,497
- 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
- 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,191
- 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
- 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
- 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,242
- 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,195
- 2025A-MEM: Agentic Memory for LLM Agentscited by 1,063
- 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agentscited by 610
31 more
- 2024Agent Workflow Memorycited by 383
- 2024Aguvis: Unified Pure Vision Agents for Autonomous GUI Interactioncited by 278
- 2024Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scalecited by 223
- 2019MineRL: A Large-Scale Dataset of Minecraft Demonstrationscited by 215
- 2024OS-Copilot: Towards Generalist Computer Agents with Self-Improvementcited by 205
- 2025SkillWeaver: Web Agents can Self-Improve by Discovering and Honing Skillscited by 192
- 2025An Illusion of Progress? Assessing the Current State of Web Agentscited by 174
- 2023JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal Language Modelscited by 171
- 2025Agent S2: A Compositional Generalist-Specialist Framework for Computer Use Agentscited by 159
- 2024Cradle: Empowering Foundation Agents Towards General Computer Controlcited by 117
- 2025AgentRewardBench: Evaluating Automatic Evaluations of Web Agent Trajectoriescited by 108
- 2023STEVE-1: A Generative Model for Text-to-Behavior in Minecraftcited by 97
- 2024Optimus-1: Hybrid Multimodal Memory Empowered Agents Excel in Long-Horizon Taskscited by 84
- 2025Contextual Experience Replay for Self-Improvement of Language Agentscited by 48
- 2026MiniMax Sparse Attentioncited by 44
- 2025Web-Shepherd: Advancing PRMs for Reinforcing Web Agentscited by 41
- 2025Advancing Mobile GUI Agents: A Verifier-Driven Approach to Practical Deploymentcited by 38
- 2025GUI-Shepherd: Reliable Process Reward and Verification for Long-Sequence GUI Taskscited by 26
- 2026CUA-Gym: Scaling Verifiable Training Environments and Tasks for Computer-Use Agentscited by 23
- 2025CUARewardBench: A Benchmark for Evaluating Reward Models on Computer-using Agentcited by 18
- 2025Scaling Agents for Computer Usecited by 14
- 2026Don't Act Blindly: Robust GUI Automation via Action-Effect Verification and Self-Correctioncited by 14
- 2026The Art of Building Verifiers for Computer Use Agentscited by 13
- 2026ToolCUA: Towards Optimal GUI-Tool Path Orchestration for Computer Use Agentscited by 11
- 2026OpenComputer: Verifiable Software Worlds for Computer-Use Agentscited by 10
- 2024MrSteve: Instruction-Following Agents in Minecraft with What-Where-When Memorycited by 9
- 2025GUI-PRA: Process Reward Agent for GUI Taskscited by 9
- 2026MAGNET: Towards Adaptive GUI Agents with Memory-Driven Knowledge Evolutioncited by 9
- 2026Darwinian Memory: A Training-Free Self-Regulating Memory System for GUI Agent Evolutioncited by 7
- 2025VerificAgent: Domain-Specific Memory Verification for Scalable Oversight of Aligned Computer-Use Agentscited by 3
- 2025"Are We Done Yet?": A Vision-Based Judge for Autonomous Task Completion of Computer Use Agentscited by 3
Built on by
Citing papers per year: 2026: 7.
7 papers.
- 2026Recursive Experiential-Working Memory Evolution for Long-Horizon Agent Harnessescited by 9
- 2026RSIAgent: Autonomous Exploration for Recursive Self-improvement in New Environmentscited by 8
- 2026RSIGame: Autonomous Agentic Game Development with Recursive Self-improvementcited by 1
- 2026Turbo Harness: Instance-Adaptive Harness Optimizationcited by 0
- 2026CUAWright: A Minimal Unified Interface for Digital Agentscited by 0
- 2026RippleCP: Measuring Counterfactual Checkpoint Advantage in Coding Agentscited by 0
- 2026Pack Light, CarryOn: Building Object-Oriented Context for Long-Horizon Agentscited by 0
- 2026Pack Light, CarryOn: Building Object-Oriented Context for Long-Horizon Agents13 Oct 2026
- 2026RippleCP: Measuring Counterfactual Checkpoint Advantage in Coding Agents6 Oct 2026
- 2026CUAWright: A Minimal Unified Interface for Digital Agents2 Oct 2026
- 2026Turbo Harness: Instance-Adaptive Harness Optimization30 Sep 2026
- 2026RSIGame: Autonomous Agentic Game Development with Recursive Self-improvement30 Sep 2026
- 2026RSIAgent: Autonomous Exploration for Recursive Self-improvement in New Environments14 Sep 2026
- 2026Recursive Experiential-Working Memory Evolution for Long-Horizon Agent Harnesses25 Aug 2026
Similar, not cited
Close in wording, no citation either way
- 2026Interlock: From Local Correctness to Task Completion in Long-Horizon Agents
- 2026Beyond Semantic Organization: Memory as Execution State Management for Long-Horizon Agents
- 2026LongHorizon-Harness: Advancing Long-Horizon Agents for Real-World Tasks
- 2026InfiAgent: An Infinite-Horizon Framework for General-Purpose Autonomous Agents
- 2026The Horizon Gap: Planning, Memory, Execution, Training, and Evaluation for Long-Horizon LLM Agents
- 2026Long-Horizon-Terminal-Bench 2.0: Automating Diverse Task Construction and Measuring Sustained Agent Progress
- 2026Staying on Task: Testing the Foundations of Long-Horizon Agent Reliability
- 2026Beyond Memory: Harnessing Long-Horizon Agents with Explicit Belief States