# Retrieval-augmented GUI Agents with Generative Guidelines arXiv: https://arxiv.org/abs/2509.24183 ยท HTML: https://mlsanity.com/paper/2509.24183/ ## Key facts - Cited by 15 papers in the map (data through 14 Oct 2026). - Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. - Builds on 24 of its 32 references that are in the map. - First posted on arXiv 29 Sep 2025. ## Placed in - Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/) - Retrieval-augmented generation / [Agentic RAG, agents and memory](https://mlsanity.com/rag/agentic-rag-agents-and-memory/) - LLM agents and tool use / [Web, search, and research agents](https://mlsanity.com/agents/web-search-and-research-agents/) ## Builds on (most cited first, 24 of 24) - [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020) - [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023) - [Text Embeddings by Weakly-Supervised Contrastive Pre-training](https://mlsanity.com/paper/2212.03533/) (2022) - [Mind2Web: Towards a Generalist Agent for the Web](https://mlsanity.com/paper/2306.06070/) (2023) - [OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments](https://mlsanity.com/paper/2404.07972/) (2024) - [REPLUG: Retrieval-Augmented Black-Box Language Models](https://mlsanity.com/paper/2301.12652/) (2023) - [SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents](https://mlsanity.com/paper/2401.10935/) (2024) - [GPT-4V(ision) is a Generalist Web Agent, if Grounded](https://mlsanity.com/paper/2401.01614/) (2024) - [Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents](https://mlsanity.com/paper/2410.05243/) (2024) - [A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis](https://mlsanity.com/paper/2307.12856/) (2023) - [FastText.zip: Compressing text classification models](https://mlsanity.com/paper/1612.03651/) (2016) - [Aguvis: Unified Pure Vision Agents for Autonomous GUI Interaction](https://mlsanity.com/paper/2412.04454/) (2024) - [RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMs](https://mlsanity.com/paper/2407.02485/) (2024) - [OmniParser for Pure Vision Based GUI Agent](https://mlsanity.com/paper/2408.00203/) (2024) - [On the Effects of Data Scale on UI Control Agents](https://mlsanity.com/paper/2406.03679/) (2024) - [Chain-of-Note: Enhancing Robustness in Retrieval-Augmented Language Models](https://mlsanity.com/paper/2311.09210/) (2023) - [AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web Tutorials](https://mlsanity.com/paper/2412.09605/) (2024) - [AMEX: Android Multi-annotation Expo Dataset for Mobile GUI Agents](https://mlsanity.com/paper/2407.17490/) (2024) - [OmniCorpus: A Unified Multimodal Corpus of 10 Billion-Level Images Interleaved with Text](https://mlsanity.com/paper/2406.08418/) (2024) - [MINT-1T: Scaling Open-Source Multimodal Data by 10x: A Multimodal Dataset with One Trillion Tokens](https://mlsanity.com/paper/2406.11271/) (2024) - [Synatra: Turning Indirect Knowledge into Direct Demonstrations for Digital Agents at Scale](https://mlsanity.com/paper/2409.15637/) (2024) - [TongUI: Internet-Scale Trajectories from Multimodal Web Tutorials for Generalized GUI Agents](https://mlsanity.com/paper/2504.12679/) (2025) - [SimRAG: Self-Improving Retrieval-Augmented Generation for Adapting Large Language Models to Specialized Domains](https://mlsanity.com/paper/2410.17952/) (2024) - [RoseRAG: Robust Retrieval-augmented Generation with Small-scale LLMs via Margin-aware Preference Optimization](https://mlsanity.com/paper/2502.10993/) (2025) ## Built on by (most cited first, 15 of 15) - [OS-Symphony: A Holistic Framework for Robust and Generalist Computer-Using Agent](https://mlsanity.com/paper/2601.07779/) (2026) - [EchoTrail-GUI: Building Actionable Memory for GUI Agents via Critic-Guided Self-Exploration](https://mlsanity.com/paper/2512.19396/) (2025) - [ContractSkill: Repairable Contract-Based Skills for Multimodal Web Agents](https://mlsanity.com/paper/2603.20340/) (2026) - [MementoGUI: Learning Agentic Multimodal Memory Control for Long-Horizon GUI Agents](https://mlsanity.com/paper/2605.18652/) (2026) - [GUIDE: Resolving Domain Bias in GUI Agents through Real-Time Web Video Retrieval and Plug-and-Play Annotation](https://mlsanity.com/paper/2603.26266/) (2026) - [UI-Copilot: Advancing Long-Horizon GUI Automation via Tool-Integrated Policy Optimization](https://mlsanity.com/paper/2604.13822/) (2026) - [MacAgentBench: Benchmarking AI Agents on Real-World macOS Desktop](https://mlsanity.com/paper/2606.22557/) (2026) - [cotomi Act: Learning to Automate Work by Watching You](https://mlsanity.com/paper/2605.03231/) (2026) - [DocOS: Towards Proactive Document-Guided Actions in GUI Agents](https://mlsanity.com/paper/2605.18048/) (2026) - [A History-Aware Visually Grounded Critic for Computer Use Agents](https://mlsanity.com/paper/2606.11078/) (2026) - [M2-Miner: Multi-Agent Enhanced MCTS for Mobile GUI Agent Data Mining](https://mlsanity.com/paper/2602.05429/) (2026) - [Weasel: Out-of-Domain Generalization for Web Agents via Importance-Diversity Data Selection](https://mlsanity.com/paper/2605.20291/) (2026) - [SynWeaver: Website-Prior Task and Trajectory Co-Synthesis for Web Agents](https://mlsanity.com/paper/2608.12429/) (2026) - [PhoneCLI: From App Interfaces to Callable Commands for Mobile Agents](https://mlsanity.com/paper/2609.35671/) (2026) - [Learning How to Remember: Memory Modality Selection for GUI Agents](https://mlsanity.com/paper/2610.17928/) (2026) ## Abstract (arXiv) GUI agents powered by vision-language models (VLMs) show promise in automating complex digital tasks. However, their effectiveness in real-world applications is often limited by scarce training data and the inherent complexity of these tasks, which frequently require long-tailed knowledge covering rare, unseen scenarios. We propose RAG-GUI , a lightweight VLM that leverages web tutorials at inference time. RAG-GUI is first warm-started via supervised finetuning (SFT) and further refined through self-guided rejection sampling finetuning (RSF). Designed to be model-agnostic, RAG-GUI functions as a generic plug-in that enhances any VLM-based agent. Evaluated across three distinct tasks, it consistently outperforms baseline agents and surpasses other inference baselines by 2.6% to 13.3% across two model sizes, demonstrating strong generalization and practical plug-and-play capabilities in real-world scenarios.