# The BrowserGym Ecosystem for Web Agent Research

arXiv: https://arxiv.org/abs/2412.05467 · HTML: https://mlsanity.com/paper/2412.05467/

## Key facts

- Cited by 161 papers in the map (data through 14 Oct 2026).
- Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use.
- Builds on 50 of its 63 references that are in the map.
- First posted on arXiv 6 Dec 2024.

## Placed in

- LLM agents and tool use / [Web, search, and research agents](https://mlsanity.com/agents/web-search-and-research-agents/)

## Builds on (most cited first, 30 of 50)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023)
- [WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents](https://mlsanity.com/paper/2207.01206/) (2022)
- [Mind2Web: Towards a Generalist Agent for the Web](https://mlsanity.com/paper/2306.06070/) (2023)
- [OpenHands: An Open Platform for AI Software Developers as Generalist Agents](https://mlsanity.com/paper/2407.16741/) (2024)
- [Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models](https://mlsanity.com/paper/2310.04406/) (2023)
- [SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents](https://mlsanity.com/paper/2401.10935/) (2024)
- [GPT-4V(ision) is a Generalist Web Agent, if Grounded](https://mlsanity.com/paper/2401.01614/) (2024)
- [Ray: A Distributed Framework for Emerging AI Applications](https://mlsanity.com/paper/1712.05889/) (2017)
- [WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models](https://mlsanity.com/paper/2401.13919/) (2024)
- [Understanding the planning of LLM agents: A survey](https://mlsanity.com/paper/2402.02716/) (2024)
- [Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents](https://mlsanity.com/paper/2410.05243/) (2024)
- [Language Models can Solve Computer Tasks](https://mlsanity.com/paper/2303.17491/) (2023)
- [WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?](https://mlsanity.com/paper/2403.07718/) (2024)
- [A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis](https://mlsanity.com/paper/2307.12856/) (2023)
- [AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents](https://mlsanity.com/paper/2401.13178/) (2024)
- [Android in the Wild: A Large-Scale Dataset for Android Device Control](https://mlsanity.com/paper/2307.10088/) (2023)
- [Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V](https://mlsanity.com/paper/2310.11441/) (2023)
- [Reinforcement Learning on Web Interfaces Using Workflow-Guided Exploration](https://mlsanity.com/paper/1802.08802/) (2018)
- [Auto-GPT for Online Decision Making: Benchmarks and Additional Opinions](https://mlsanity.com/paper/2306.02224/) (2023)
- [ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddings](https://mlsanity.com/paper/2305.11554/) (2023)
- [Synapse: Trajectory-as-Exemplar Prompting with Memory for Computer Control](https://mlsanity.com/paper/2306.07863/) (2023)
- [Mapping Natural Language Instructions to Mobile UI Action Sequences](https://mlsanity.com/paper/2005.03776/) (2020)
- [WebLINX: Real-World Website Navigation with Multi-Turn Dialogue](https://mlsanity.com/paper/2402.05930/) (2024)
- [Multimodal Web Navigation with Instruction-Finetuned Foundation Models](https://mlsanity.com/paper/2305.11854/) (2023)
- [AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks?](https://mlsanity.com/paper/2407.15711/) (2024)

## Built on by (most cited first, 30 of 161)

- [TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks](https://mlsanity.com/paper/2412.14161/) (2024)
- [ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memory](https://mlsanity.com/paper/2509.25140/) (2025)
- [Survey on Evaluation of LLM-based Agents](https://mlsanity.com/paper/2503.16416/) (2025)
- [Inducing Programmatic Skills for Agentic Tasks](https://mlsanity.com/paper/2504.06821/) (2025)
- [SafeArena: Evaluating the Safety of Autonomous Web Agents](https://mlsanity.com/paper/2503.04957/) (2025)
- [Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation](https://mlsanity.com/paper/2510.11977/) (2025)
- [A Survey of WebAgents: Towards Next-Generation AI Agents for Web Automation with Large Foundation Models](https://mlsanity.com/paper/2503.23350/) (2025)
- [ComputerRL: Scaling End-to-End Online Reinforcement Learning for Computer Use Agents](https://mlsanity.com/paper/2508.14040/) (2025)
- [RedTeamCUA: Realistic Adversarial Testing of Computer-Use Agents in Hybrid Web-OS Environments](https://mlsanity.com/paper/2505.21936/) (2025)
- [Web-Shepherd: Advancing PRMs for Reinforcing Web Agents](https://mlsanity.com/paper/2505.15277/) (2025)
- [OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks](https://mlsanity.com/paper/2606.29537/) (2026)
- [CowPilot: A Framework for Autonomous and Human-Agent Collaborative Web Navigation](https://mlsanity.com/paper/2501.16609/) (2025)
- [A Survey of Large Language Model Empowered Agents for Recommendation and Search: Towards Next-Generation Information Retrieval](https://mlsanity.com/paper/2503.05659/) (2025)
- [Go-Browse: Training Web Agents with Structured Exploration](https://mlsanity.com/paper/2506.03533/) (2025)
- [DoomArena: A framework for Testing AI Agents Against Evolving Security Threats](https://mlsanity.com/paper/2504.14064/) (2025)
- [APEX-Agents](https://mlsanity.com/paper/2601.14242/) (2026)
- [WebChoreArena: Evaluating Web Browsing Agents on Realistic Tedious Web Tasks](https://mlsanity.com/paper/2506.01952/) (2025)
- [Agentic AI Security: Threats, Defenses, Evaluation, and Open Challenges](https://mlsanity.com/paper/2510.23883/) (2025)
- [PolySkill: Learning Generalizable Skills Through Polymorphic Abstraction](https://mlsanity.com/paper/2510.15863/) (2025)
- [Coding Agents with Multimodal Browsing are Generalist Problem Solvers](https://mlsanity.com/paper/2506.03011/) (2025)
- [Symbiotic Cooperation for Web Agents: Harnessing Complementary Strengths of Large and Small LLMs](https://mlsanity.com/paper/2502.07942/) (2025)
- [MedAgentGym: A Scalable Agentic Training Environment for Code-Centric Reasoning in Biomedical Data Science](https://mlsanity.com/paper/2506.04405/) (2025)
- [WebGuard: Building a Generalizable Guardrail for Web Agents](https://mlsanity.com/paper/2507.14293/) (2025)
- [AgentA/B: Automated and Scalable Web A/BTesting with Interactive LLM Agents](https://mlsanity.com/paper/2504.09723/) (2025)
- [How to Train Your LLM Web Agent: A Statistical Diagnosis](https://mlsanity.com/paper/2507.04103/) (2025)
- [MolmoWeb: Open Visual Web Agent and Open Data for the Open Web](https://mlsanity.com/paper/2604.08516/) (2026)
- [Inside the Scaffold: A Source-Code Taxonomy of Coding Agent Architectures](https://mlsanity.com/paper/2604.03515/) (2026)
- [WebMall -- A Multi-Shop Benchmark for Evaluating Web Agents](https://mlsanity.com/paper/2508.13024/) (2025)
- [BrowserAgent: Building Web Agents with Human-Inspired Web Browsing Actions](https://mlsanity.com/paper/2510.10666/) (2025)
- [Agent Data Protocol: Unifying Datasets for Diverse, Effective Fine-tuning of LLM Agents](https://mlsanity.com/paper/2510.24702/) (2025)

## Abstract (arXiv)

The BrowserGym ecosystem addresses the growing need for efficient evaluation and benchmarking of web agents, particularly those leveraging automation and Large Language Models (LLMs). Many existing benchmarks suffer from fragmentation and inconsistent evaluation methodologies, making it challenging to achieve reliable comparisons and reproducible results. In an earlier work, Drouin et al. (2024) introduced BrowserGym which aims to solve this by providing a unified, gym-like environment with well-defined observation and action spaces, facilitating standardized evaluation across diverse benchmarks. We propose an extended BrowserGym-based ecosystem for web agent research, which unifies existing benchmarks from the literature and includes AgentLab, a complementary framework that aids in agent creation, testing, and analysis. Our proposed ecosystem offers flexibility for integrating new benchmarks while ensuring consistent evaluation and comprehensive experiment management. As a supporting evidence, we conduct the first large-scale, multi-benchmark web agent experiment and compare the performance of 6 state-of-the-art LLMs across 6 popular web agent benchmarks made available in BrowserGym. Among other findings, our results highlight a large discrepancy between OpenAI and Anthropic's latests models, with Claude-3.5-Sonnet leading the way on almost all benchmarks, except on vision-related tasks where GPT-4o is superior. Despite these advancements, our results emphasize that building robust and efficient web agents remains a significant challenge, due to the inherent complexity of real-world web environments and the limitations of current models.