# WebFactory: Automated Compression of Foundational Language Intelligence into Grounded Web Agents

arXiv: https://arxiv.org/abs/2603.05044 · HTML: https://mlsanity.com/paper/2603.05044/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use.
- Builds on 39 of its 44 references that are in the map.
- First posted on arXiv 5 Mar 2026.

## Placed in

- LLM agents and tool use / [Web, search, and research agents](https://mlsanity.com/agents/web-search-and-research-agents/)

## Builds on (most cited first, 30 of 39)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [Voyager: An Open-Ended Embodied Agent with Large Language Models](https://mlsanity.com/paper/2305.16291/) (2023)
- [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://mlsanity.com/paper/2106.01345/) (2021)
- [Offline Reinforcement Learning with Implicit Q-Learning](https://mlsanity.com/paper/2110.06169/) (2021)
- [WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents](https://mlsanity.com/paper/2207.01206/) (2022)
- [Mind2Web: Towards a Generalist Agent for the Web](https://mlsanity.com/paper/2306.06070/) (2023)
- [ALFWorld: Aligning Text and Embodied Environments for Interactive Learning](https://mlsanity.com/paper/2010.03768/) (2020)
- [WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models](https://mlsanity.com/paper/2401.13919/) (2024)
- [Android in the Wild: A Large-Scale Dataset for Android Device Control](https://mlsanity.com/paper/2307.10088/) (2023)
- [Reinforcement Learning on Web Interfaces Using Workflow-Guided Exploration](https://mlsanity.com/paper/1802.08802/) (2018)
- [WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learning](https://mlsanity.com/paper/2411.02337/) (2024)
- [BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learning](https://mlsanity.com/paper/1810.08272/) (2018)
- [GUI-R1 : A Generalist R1-Style Vision-Language Action Model For GUI Agents](https://mlsanity.com/paper/2504.10458/) (2025)
- [Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale](https://mlsanity.com/paper/2409.08264/) (2024)
- [GUIOdyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devices](https://mlsanity.com/paper/2406.08451/) (2024)
- [SkillWeaver: Web Agents can Self-Improve by Discovering and Honing Skills](https://mlsanity.com/paper/2504.07079/) (2025)
- [WebAgent-R1: Training Web Agents via End-to-End Multi-Turn Reinforcement Learning](https://mlsanity.com/paper/2505.16421/) (2025)
- [The BrowserGym Ecosystem for Web Agent Research](https://mlsanity.com/paper/2412.05467/) (2024)
- [Multimodal Web Navigation with Instruction-Finetuned Foundation Models](https://mlsanity.com/paper/2305.11854/) (2023)
- [OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Web](https://mlsanity.com/paper/2402.17553/) (2024)
- [AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web Tutorials](https://mlsanity.com/paper/2412.09605/) (2024)
- [WebCanvas: Benchmarking Web Agents in Online Environments](https://mlsanity.com/paper/2406.12373/) (2024)
- [GUICourse: From General Vision Language Models to Versatile GUI Agents](https://mlsanity.com/paper/2406.11317/) (2024)
- [AgentOccam: A Simple Yet Strong Baseline for LLM-Based Web Agents](https://mlsanity.com/paper/2410.13825/) (2024)
- [Explorer: Scaling Exploration-driven Web Trajectory Synthesis for Multimodal Web Agents](https://mlsanity.com/paper/2502.11357/) (2025)

## Built on by (most cited first, 6 of 6)

- [Qwen-AgentWorld: Language World Models for General Agents](https://mlsanity.com/paper/2606.24597/) (2026)
- [Learn where to Click from Yourself: On-Policy Self-Distillation for GUI Grounding](https://mlsanity.com/paper/2605.00642/) (2026)
- [Qwen-CUA: Native Computer Use for (almost) Everything](https://mlsanity.com/paper/2608.02352/) (2026)
- [Fara-1.5: Scalable Learning Environments for Computer Use Agents](https://mlsanity.com/paper/2606.20785/) (2026)
- [EvoCUA-1.5: Online Reinforcement Learning for Multi-turn Computer-Use Agents](https://mlsanity.com/paper/2607.09773/) (2026)
- [From Trajectories to Grounded Preferences: Process Preference Synthesis via Interaction Element Graphs for Web PRMs](https://mlsanity.com/paper/2609.32351/) (2026)

## Abstract (arXiv)

Current paradigms for training GUI agents are fundamentally limited by a reliance on either unsafe, non-reproducible live web interactions or costly, scarce human-crafted data and environments. We argue this focus on data volume overlooks a more critical factor: the efficiency of compressing a large language model's (LLM) latent knowledge into actionable agent behavior. We introduce WebFactory, a novel, fully automated closed-loop reinforcement learning pipeline for GUI agents, systematically compressing LLM-encoded internet intelligence into efficient, grounded actions. Our pipeline features a process of scalable environment synthesis, knowledge-aware task generation, LLM-powered trajectory collection, decomposed reward RL training, and systematic agent evaluation. Remarkably, our agent demonstrates exceptional data efficiency and generalization. Trained on synthetic data from only 10 websites within WebFactory, it achieves performance comparable to GUI agents trained on the same amount of human-annotated data from a much larger set of environments. This superior performance is consistent across our internal offline and online transfer benchmarks, where our agent also significantly outperforms the base foundation model. We further provide critical insights into the "embodiment potential" of different LLM foundations, offering a new axis for model evaluation. This work presents a scalable and cost-effective paradigm for transforming passive internet knowledge into active, grounded intelligence, marking a critical step towards general-purpose interactive agents.