# Beyond BeautifulSoup: Benchmarking LLM-Powered Web Scraping for Everyday Users

arXiv: https://arxiv.org/abs/2601.06301 · HTML: https://mlsanity.com/paper/2601.06301/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 23 references that are in the map.
- First posted on arXiv 9 Jan 2026.

## Builds on (most cited first, 8 of 8)

- [OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments](https://mlsanity.com/paper/2404.07972/) (2024)
- [Survey on Evaluation of LLM-based Agents](https://mlsanity.com/paper/2503.16416/) (2025)
- [Tranco: A Research-Oriented Top Sites Ranking Hardened Against Manipulation](https://mlsanity.com/paper/1806.01156/) (2018)
- [Oedipus: LLM-enchanced Reasoning CAPTCHA Solver](https://mlsanity.com/paper/2405.07496/) (2024)
- [Open CaptchaWorld: A Comprehensive Web-based Platform for Testing and Benchmarking Multimodal LLM Agents](https://mlsanity.com/paper/2505.24878/) (2025)
- [IllusionCAPTCHA: A CAPTCHA based on Visual Illusion](https://mlsanity.com/paper/2502.05461/) (2025)
- [WebLists: Extracting Structured Information From Complex Interactive Websites Using Executable LLM Agents](https://mlsanity.com/paper/2504.12682/) (2025)
- [PRITES: An integrative framework for investigating and assessing web-scraped HTTP-response datasets for research applications](https://mlsanity.com/paper/2511.13773/) (2025)

## Built on by (most cited first, 1 of 1)

- [Next-Gen CAPTCHAs: Leveraging the Cognitive Gap for Scalable and Diverse GUI-Agent Defense](https://mlsanity.com/paper/2602.09012/) (2026)

## Abstract (arXiv)

Web scraping has historically required technical expertise in HTML parsing, session management, and authentication circumvention, which limited large-scale data extraction to skilled developers. We argue that large language models (LLMs) have democratized web scraping, enabling low-skill users to execute sophisticated operations through simple natural language prompts. While extensive benchmarks evaluate these tools under optimal expert conditions, we show that without extensive manual effort, current LLM-based workflows allow novice users to scrape complex websites that would otherwise be inaccessible. We systematically benchmark what everyday users can do with off-the-shelf LLM tools across 35 sites spanning five security tiers, including authentication, anti-bot, and CAPTCHA controls. We devise and evaluate two distinct workflows: (a) LLM-assisted scripting, where users prompt LLMs to generate traditional scraping code but maintain manual execution control, and (b) end-to-end LLM agents, which autonomously navigate and extract data through integrated tool use. Our results demonstrate that end-to-end agents have made complex scraping accessible - requiring as little as a single prompt with minimal refinement (less than 5 changes) to complete workflows. We also highlight scenarios where LLM-assisted scripting may be simpler and faster for static sites. In light of these findings, we provide simple procedures for novices to use these workflows and gauge what adversaries could achieve using these.