mlsanityData to 14 Oct 2026

Retrieval-augmented GUI Agents with Generative Guidelines

arXiv 2509.24183 ↗First posted 29 Sep 2025cs.CL cs.AI cs.LGcited by 15

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

also in 2 more
Abstract (arXiv)

GUI agents powered by vision-language models (VLMs) show promise in automating complex digital tasks. However, their effectiveness in real-world applications is often limited by scarce training data and the inherent complexity of these tasks, which frequently require long-tailed knowledge covering rare, unseen scenarios. We propose RAG-GUI , a lightweight VLM that leverages web tutorials at inference time. RAG-GUI is first warm-started via supervised finetuning (SFT) and further refined through self-guided rejection sampling finetuning (RSF). Designed to be model-agnostic, RAG-GUI functions as a generic plug-in that enhances any VLM-based agent. Evaluated across three distinct tasks, it consistently outperforms baseline agents and surpasses other inference baselines by 2.6% to 13.3% across two model sizes, demonstrating strong generalization and practical plug-and-play capabilities in real-world scenarios.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 24 of its 32 references that are in the map. First posted on arXiv 29 Sep 2025.

Builds on

24 of 32 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents7 Oct
  2. 2024OmniParser for Pure Vision Based GUI Agent1 Aug

Other references · 22

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  2. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
  3. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,253
  4. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,242
  5. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,195
  6. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 792
  7. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 567
  8. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 563
  9. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 353
  10. 2016FastText.zip: Compressing text classification modelscited by 335
  11. 2024Aguvis: Unified Pure Vision Agents for Autonomous GUI Interactioncited by 278
  12. 2024RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMscited by 202
10 more
  1. 2024On the Effects of Data Scale on UI Control Agentscited by 194
  2. 2023Chain-of-Note: Enhancing Robustness in Retrieval-Augmented Language Modelscited by 184
  3. 2024AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web Tutorialscited by 130
  4. 2024AMEX: Android Multi-annotation Expo Dataset for Mobile GUI Agentscited by 113
  5. 2024OmniCorpus: A Unified Multimodal Corpus of 10 Billion-Level Images Interleaved with Textcited by 59
  6. 2024MINT-1T: Scaling Open-Source Multimodal Data by 10x: A Multimodal Dataset with One Trillion Tokenscited by 50
  7. 2024Synatra: Turning Indirect Knowledge into Direct Demonstrations for Digital Agents at Scalecited by 43
  8. 2025TongUI: Internet-Scale Trajectories from Multimodal Web Tutorials for Generalized GUI Agentscited by 42
  9. 2024SimRAG: Self-Improving Retrieval-Augmented Generation for Adapting Large Language Models to Specialized Domainscited by 28
  10. 2025RoseRAG: Robust Retrieval-augmented Generation with Small-scale LLMs via Margin-aware Preference Optimizationcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 14.

15 papers.

Sort:

By area:

  1. 2026OS-Symphony: A Holistic Framework for Robust and Generalist Computer-Using Agentcited by 21
  2. 2025EchoTrail-GUI: Building Actionable Memory for GUI Agents via Critic-Guided Self-Explorationcited by 15
  3. 2026ContractSkill: Repairable Contract-Based Skills for Multimodal Web Agentscited by 13
  4. 2026MementoGUI: Learning Agentic Multimodal Memory Control for Long-Horizon GUI Agentscited by 11
  5. 2026GUIDE: Resolving Domain Bias in GUI Agents through Real-Time Web Video Retrieval and Plug-and-Play Annotationcited by 4
  6. 2026UI-Copilot: Advancing Long-Horizon GUI Automation via Tool-Integrated Policy Optimizationcited by 4
  7. 2026MacAgentBench: Benchmarking AI Agents on Real-World macOS Desktopcited by 2
  8. 2026cotomi Act: Learning to Automate Work by Watching Youcited by 1
  9. 2026DocOS: Towards Proactive Document-Guided Actions in GUI Agentscited by 1
  10. 2026A History-Aware Visually Grounded Critic for Computer Use Agentscited by 1
  11. 2026M2-Miner: Multi-Agent Enhanced MCTS for Mobile GUI Agent Data Miningcited by 0
  12. 2026Weasel: Out-of-Domain Generalization for Web Agents via Importance-Diversity Data Selectioncited by 0
3 more
  1. 2026SynWeaver: Website-Prior Task and Trajectory Co-Synthesis for Web Agentscited by 0
  2. 2026PhoneCLI: From App Interfaces to Callable Commands for Mobile Agentscited by 0
  3. 2026Learning How to Remember: Memory Modality Selection for GUI Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Towards Agentic RAG with Deep Reasoning: A Survey of RAG-Reasoning Systems in LLMs
  2. 2025Breaking the Data Barrier -- Building GUI Agents Through Task Generalization
  3. 2025UI-AGILE: Advancing GUI Agents with Effective Reinforcement Learning and Precise Inference-Time Grounding
  4. 2026RAGShaper: Eliciting Sophisticated Agentic RAG Skills via Automated Data Synthesis
  5. 2025Interact-RAG: Reason and Interact with the Corpus, Beyond Black-Box Retrieval
  6. 2024RAG-DDR: Optimizing Retrieval-Augmented Generation Using Differentiable Data Rewards
  7. 2024EDGE: Enhanced Grounded GUI Understanding with Enriched Multi-Granularity Synthetic Data
  8. 2026A-RAG: Scaling Agentic Retrieval-Augmented Generation via Hierarchical Retrieval Interfaces