# Hybrid Self-evolving Structured Memory for GUI Agents

arXiv: https://arxiv.org/abs/2603.10291 · HTML: https://mlsanity.com/paper/2603.10291/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 15 of its 31 references that are in the map.
- First posted on arXiv 11 Mar 2026.

## Placed in

- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)
- LLM agents and tool use / [Agent memory systems](https://mlsanity.com/agents/agent-memory-systems/)
- Synthetic data and self-improvement for LLMs / [Self-evolving LLM agents](https://mlsanity.com/synth/self-evolving-llm-agents/)

## Builds on (most cited first, 15 of 15)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [MemGPT: Towards LLMs as Operating Systems](https://mlsanity.com/paper/2310.08560/) (2023)
- [Mind2Web: Towards a Generalist Agent for the Web](https://mlsanity.com/paper/2306.06070/) (2023)
- [A-MEM: Agentic Memory for LLM Agents](https://mlsanity.com/paper/2502.12110/) (2025)
- [UI-TARS: Pioneering Automated GUI Interaction with Native Agents](https://mlsanity.com/paper/2501.12326/) (2025)
- [GPT-4V(ision) is a Generalist Web Agent, if Grounded](https://mlsanity.com/paper/2401.01614/) (2024)
- [AppAgent: Multimodal Agents as Smartphone Users](https://mlsanity.com/paper/2312.13771/) (2023)
- [Memory in the Age of AI Agents](https://mlsanity.com/paper/2512.13564/) (2025)
- [GUICourse: From General Vision Language Models to Versatile GUI Agents](https://mlsanity.com/paper/2406.11317/) (2024)
- [MemGen: Weaving Generative Latent Memory for Self-Evolving Agents](https://mlsanity.com/paper/2509.24704/) (2025)
- [MMInA: Benchmarking Multihop Multimodal Internet Agents](https://mlsanity.com/paper/2404.09992/) (2024)
- [Visual Test-time Scaling for GUI Agent Grounding](https://mlsanity.com/paper/2505.00684/) (2025)
- [Auto-scaling Continuous Memory for GUI Agent](https://mlsanity.com/paper/2510.09038/) (2025)
- [ProBench: Benchmarking GUI Agents with Accurate Process Information](https://mlsanity.com/paper/2511.09157/) (2025)

## Built on by (most cited first, 13 of 13)

- [MementoGUI: Learning Agentic Multimodal Memory Control for Long-Horizon GUI Agents](https://mlsanity.com/paper/2605.18652/) (2026)
- [StructAgent: Harness Long-horizon Digital Agents with Unified Causal Structure](https://mlsanity.com/paper/2607.11388/) (2026)
- [What Memory Do GUI Agents Really Need? From Passive Records to Active Task-Driving States](https://mlsanity.com/paper/2606.31612/) (2026)
- [Planner Matters! An Efficient and Unbalanced Multi-agent Collaboration Framework for Long-horizon Planning](https://mlsanity.com/paper/2605.02168/) (2026)
- [Mem-W: Latent Memory-Native GUI Agents](https://mlsanity.com/paper/2605.09317/) (2026)
- [DMV-Bench: Diagnosing Long-Horizon Multimodal Agents' Visual Memory with Incidental Cue Injection](https://mlsanity.com/paper/2606.27499/) (2026)
- [FocusMem: Factorizing Content, Readout, and Trust in Latent GUI Memory](https://mlsanity.com/paper/2608.04530/) (2026)
- [Proactive Service Agents: A Unified Decision Framework, Methods, and Evaluation](https://mlsanity.com/paper/2609.03727/) (2026)
- [Reflect, Revise, Reuse: Training-Free Skill Evolution for GUI Agents](https://mlsanity.com/paper/2609.17653/) (2026)
- [Learning to Learn from Multimodal Experience](https://mlsanity.com/paper/2605.16857/) (2026)
- [Marginal Advantage Accumulation for Memory-Driven Agent Self-Evolution](https://mlsanity.com/paper/2606.20475/) (2026)
- [EchoPath: Execution-Level Replayable Memory for GUI Agents](https://mlsanity.com/paper/2609.16635/) (2026)
- [Relevance Does Not Imply Applicability: Experience Activation for Personal GUI Agents](https://mlsanity.com/paper/2609.33304/) (2026)

## Abstract (arXiv)

The remarkable progress of vision-language models (VLMs) has enabled GUI agents to interact with computers in a human-like manner. Yet real-world computer-use tasks remain difficult due to long-horizon workflows, diverse interfaces, and frequent intermediate errors. Prior work equips agents with external memory built from large collections of trajectories, but relies on flat retrieval over discrete summaries or continuous embeddings, falling short of the structured organization and self-evolving characteristics of human memory. Inspired by the brain, we propose Hybrid Self-evolving Structured Memory (HyMEM), a graph-based memory that couples discrete high-level symbolic nodes with continuous trajectory embeddings. HyMEM maintains a graph structure to support multi-hop retrieval, self-evolution via node update operations, and on-the-fly working-memory refreshing during inference. Extensive experiments show that HyMEM consistently improves open-source GUI agents, enabling 7B/8B backbones to match or surpass strong closed-source models; notably, it boosts Qwen2.5-VL-7B by +22.5% and outperforms Gemini2.5-Pro-Vision and GPT-4o.