# WebLists: Extracting Structured Information From Complex Interactive Websites Using Executable LLM Agents

arXiv: https://arxiv.org/abs/2504.12682 · HTML: https://mlsanity.com/paper/2504.12682/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use.
- Builds on 15 of its 35 references that are in the map.
- First posted on arXiv 17 Apr 2025.

## Placed in

- LLM agents and tool use / [Web, search, and research agents](https://mlsanity.com/agents/web-search-and-research-agents/)

## Builds on (most cited first, 15 of 15)

- [WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents](https://mlsanity.com/paper/2207.01206/) (2022)
- [Mind2Web: Towards a Generalist Agent for the Web](https://mlsanity.com/paper/2306.06070/) (2023)
- [Language Models can Solve Computer Tasks](https://mlsanity.com/paper/2303.17491/) (2023)
- [Agent S: An Open Agentic Framework that Uses Computers Like a Human](https://mlsanity.com/paper/2410.08164/) (2024)
- [Is Your LLM Secretly a World Model of the Internet? Model-Based Planning for Web Agents](https://mlsanity.com/paper/2411.06559/) (2024)
- [AgentOccam: A Simple Yet Strong Baseline for LLM-Based Web Agents](https://mlsanity.com/paper/2410.13825/) (2024)
- [Agent-E: From Autonomous Web Navigation to Foundational Design Principles in Agentic Systems](https://mlsanity.com/paper/2407.13032/) (2024)
- [Large Language Models Empowered Personalized Web Agents](https://mlsanity.com/paper/2410.17236/) (2024)
- [OpenWebVoyager: Building Multimodal Web Agents via Iterative Real-World Exploration, Feedback and Optimization](https://mlsanity.com/paper/2410.19609/) (2024)
- [BEARCUBS: A benchmark for computer-using web agents](https://mlsanity.com/paper/2503.07919/) (2025)
- [WILBUR: Adaptive In-Context Learning for Robust and Accurate Web Agents](https://mlsanity.com/paper/2404.05902/) (2024)
- [WebGames: Challenging General-Purpose Web-Browsing AI Agents](https://mlsanity.com/paper/2502.18356/) (2025)
- [Web2Text: Deep Structured Boilerplate Removal](https://mlsanity.com/paper/1801.02607/) (2018)
- [Fundus: A Simple-to-Use News Scraper Optimized for High Quality Extractions](https://mlsanity.com/paper/2403.15279/) (2024)
- [Multilingual Attribute Extraction from News Web Pages](https://mlsanity.com/paper/2502.02167/) (2025)

## Built on by (most cited first, 3 of 3)

- [Semantics Delivery Network: Rethinking Web Retrieval Infrastructure for LLM Agents](https://mlsanity.com/paper/2609.22486/) (2026)
- [Beyond BeautifulSoup: Benchmarking LLM-Powered Web Scraping for Everyday Users](https://mlsanity.com/paper/2601.06301/) (2026)
- [news-crawler-LM: A Small Long-Context Model For High-Quality News Crawling](https://mlsanity.com/paper/2607.21284/) (2026)

## Abstract (arXiv)

Most recent web agent research has focused on navigation and transaction tasks, with little emphasis on extracting structured data at scale. We present WebLists, a benchmark of 200 data-extraction tasks across four common business and enterprise use-cases. Each task requires an agent to navigate to a webpage, configure it appropriately, and extract complete datasets with well-defined schemas. We show that both LLMs with search capabilities and SOTA web agents struggle with these tasks, with a recall of 3% and 31%, respectively, despite higher performance on question-answering tasks. To address this challenge, we propose BardeenAgent, a novel framework that enables web agents to convert their execution into repeatable programs, and replay them at scale across pages with similar structure. BardeenAgent is also the first LLM agent to take advantage of the regular structure of HTML. In particular BardeenAgent constructs a generalizable CSS selector to capture all relevant items on the page, then fits the operations to extract the data. On the WebLists benchmark, BardeenAgent achieves 66% recall overall, more than doubling the performance of SOTA web agents, and reducing cost per output row by 3x.