# Build the web for agents, not agents for the web arXiv: https://arxiv.org/abs/2506.10953 ยท HTML: https://mlsanity.com/paper/2506.10953/ ## Key facts - Cited by 18 papers in the map (data through 14 Oct 2026). - Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use. - Builds on 50 of its 64 references that are in the map. - First posted on arXiv 12 Jun 2025. ## Placed in - LLM agents and tool use / [Web, search, and research agents](https://mlsanity.com/agents/web-search-and-research-agents/) ## Builds on (most cited first, 30 of 50) - [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020) - [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023) - [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023) - [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020) - [REALM: Retrieval-Augmented Language Model Pre-Training](https://mlsanity.com/paper/2002.08909/) (2020) - [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023) - [SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering](https://mlsanity.com/paper/2405.15793/) (2024) - [Mind2Web: Towards a Generalist Agent for the Web](https://mlsanity.com/paper/2306.06070/) (2023) - [OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments](https://mlsanity.com/paper/2404.07972/) (2024) - [Latent Retrieval for Weakly Supervised Open Domain Question Answering](https://mlsanity.com/paper/1906.00300/) (2019) - [UI-TARS: Pioneering Automated GUI Interaction with Native Agents](https://mlsanity.com/paper/2501.12326/) (2025) - [GPT-4V(ision) is a Generalist Web Agent, if Grounded](https://mlsanity.com/paper/2401.01614/) (2024) - [AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents](https://mlsanity.com/paper/2405.14573/) (2024) - [WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models](https://mlsanity.com/paper/2401.13919/) (2024) - [AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents](https://mlsanity.com/paper/2410.09024/) (2024) - [LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders](https://mlsanity.com/paper/2404.05961/) (2024) - [Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents](https://mlsanity.com/paper/2410.05243/) (2024) - [WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?](https://mlsanity.com/paper/2403.07718/) (2024) - [A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis](https://mlsanity.com/paper/2307.12856/) (2023) - [Android in the Wild: A Large-Scale Dataset for Android Device Control](https://mlsanity.com/paper/2307.10088/) (2023) - [Reinforcement Learning on Web Interfaces Using Workflow-Guided Exploration](https://mlsanity.com/paper/1802.08802/) (2018) - [On the Effects of Data Scale on UI Control Agents](https://mlsanity.com/paper/2406.03679/) (2024) - [SkillWeaver: Web Agents can Self-Improve by Discovering and Honing Skills](https://mlsanity.com/paper/2504.07079/) (2025) - [WebLINX: Real-World Website Navigation with Multi-Turn Dialogue](https://mlsanity.com/paper/2402.05930/) (2024) - [ADaPT: As-Needed Decomposition and Planning with Language Models](https://mlsanity.com/paper/2311.05772/) (2023) - [An Illusion of Progress? Assessing the Current State of Web Agents](https://mlsanity.com/paper/2504.01382/) (2025) - [Multimodal Web Navigation with Instruction-Finetuned Foundation Models](https://mlsanity.com/paper/2305.11854/) (2023) - [Is Your LLM Secretly a World Model of the Internet? Model-Based Planning for Web Agents](https://mlsanity.com/paper/2411.06559/) (2024) - [Tree Search for Language Model Agents](https://mlsanity.com/paper/2407.01476/) (2024) - [ST-WebAgentBench: A Benchmark for Evaluating Safety and Trustworthiness in Web Agents](https://mlsanity.com/paper/2410.06703/) (2024) ## Built on by (most cited first, 18 of 18) - [From Prompt Injections to Protocol Exploits: Threats in LLM-Powered AI Agents Workflows](https://mlsanity.com/paper/2506.23260/) (2025) - [Agentic Web: Weaving the Next Web with AI Agents](https://mlsanity.com/paper/2507.21206/) (2025) - [WALT: Web Agents that Learn Tools](https://mlsanity.com/paper/2510.01524/) (2025) - [ACE-Router: Generalizing History-Aware Routing from MCP Tools to the Agent Web](https://mlsanity.com/paper/2601.08276/) (2026) - [How do Visual Attributes Influence Web Agents? A Comprehensive Evaluation of User Interface Design Factors](https://mlsanity.com/paper/2601.21961/) (2026) - [AgentWebBench: Benchmarking Multi-Agent Coordination in Agentic Web](https://mlsanity.com/paper/2604.10938/) (2026) - [Interaction-Driven Browsing: A Human-in-the-Loop Conceptual Framework Informed by Human Web Browsing for Browser-Using Agents](https://mlsanity.com/paper/2509.12049/) (2025) - [WebRouter: Query-specific Router via Variational Information Bottleneck for Cost-sensitive Web Agent](https://mlsanity.com/paper/2510.11221/) (2025) - [Affordance Representation and Recognition for Autonomous Agents](https://mlsanity.com/paper/2510.24459/) (2025) - [Reliable agent engineering should integrate machine-compatible organizational principles](https://mlsanity.com/paper/2512.07665/) (2025) - [DeepXiv-SDK: An Agentic Data Interface for Scientific Literature](https://mlsanity.com/paper/2603.00084/) (2026) - [AgentBeats: Agentifying Agent Assessment for Openness, Standardization, and Reproducibility](https://mlsanity.com/paper/2606.13608/) (2026) - [WebSP-Eval: Evaluating Web Agents on Website Security and Privacy Tasks](https://mlsanity.com/paper/2604.06367/) (2026) - [CI4A: Semantic Component Interfaces for Agents Empowering Web Automation](https://mlsanity.com/paper/2601.14790/) (2026) - [Internal APIs Are All You Need: Shadow APIs, Shared Discovery, and the Case Against Browser-First Agent Architectures](https://mlsanity.com/paper/2604.00694/) (2026) - [SW-A2-Bench: Benchmarking Autonomous Software Agent Generation for Agentic Web](https://mlsanity.com/paper/2604.04226/) (2026) - [AnyAct: Universal Action for Self-Evolving Agents](https://mlsanity.com/paper/2609.37025/) (2026) - [Get In, Decide, Prove: A Merchant-Side Readiness Benchmark for Agentic Commerce](https://mlsanity.com/paper/2610.17378/) (2026) ## Abstract (arXiv) Recent advancements in Large Language Models (LLMs) and multimodal counterparts have spurred significant interest in developing web agents -- AI systems capable of autonomously navigating and completing tasks within web environments. While holding tremendous promise for automating complex web interactions, current approaches face substantial challenges due to the fundamental mismatch between human-designed interfaces and LLM capabilities. Current methods struggle with the inherent complexity of web inputs, whether processing massive DOM trees, relying on screenshots augmented with additional information, or bypassing the user interface entirely through API interactions. This position paper advocates for a paradigm shift in web agent research: rather than forcing web agents to adapt to interfaces designed for humans, we should develop a new interaction paradigm specifically optimized for agentic capabilities. To this end, we introduce the concept of an Agentic Web Interface (AWI), an interface specifically designed for agents to navigate a website. We establish six guiding principles for AWI design, emphasizing safety, efficiency, and standardization, to account for the interests of all primary stakeholders. This reframing aims to overcome fundamental limitations of existing interfaces, paving the way for more efficient, reliable, and transparent web agent design, which will be a collaborative effort involving the broader ML community.