# AutoScraper: A Progressive Understanding Web Agent for Web Scraper Generation

arXiv: https://arxiv.org/abs/2404.12753 · HTML: https://mlsanity.com/paper/2404.12753/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use.
- Builds on 17 of its 36 references that are in the map.
- First posted on arXiv 19 Apr 2024.

## Placed in

- LLM agents and tool use / [Web, search, and research agents](https://mlsanity.com/agents/web-search-and-research-agents/)

## Builds on (most cited first, 17 of 17)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents](https://mlsanity.com/paper/2207.01206/) (2022)
- [DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence](https://mlsanity.com/paper/2401.14196/) (2024)
- [Teaching Large Language Models to Self-Debug](https://mlsanity.com/paper/2304.05128/) (2023)
- [A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis](https://mlsanity.com/paper/2307.12856/) (2023)
- [Synapse: Trajectory-as-Exemplar Prompting with Memory for Computer Control](https://mlsanity.com/paper/2306.07863/) (2023)
- [LASER: LLM Agent with State-Space Exploration for Web Navigation](https://mlsanity.com/paper/2309.08172/) (2023)
- [MarkupLM: Pre-training of Text and Markup Language for Visually-rich Document Understanding](https://mlsanity.com/paper/2110.08518/) (2021)
- [Hierarchical Prompting Assists Large Language Model on Web Navigation](https://mlsanity.com/paper/2305.14257/) (2023)
- [WebFormer: The Web-page Transformer for Structure Information Extraction](https://mlsanity.com/paper/2202.00217/) (2022)
- [FreeDOM: A Transferable Neural Architecture for Structured Information Extraction on Web Documents](https://mlsanity.com/paper/2010.10755/) (2020)
- [Simplified DOM Trees for Transferable Attribute Extraction from the Web](https://mlsanity.com/paper/2101.02415/) (2021)
- [CERES: Distantly Supervised Relation Extraction from the Semi-Structured Web](https://mlsanity.com/paper/1804.04635/) (2018)
- [WebIE: Faithful and Robust Information Extraction on the Web](https://mlsanity.com/paper/2305.14293/) (2023)

## Built on by (most cited first, 20 of 20)

- [AutoData: A Multi-Agent System for Open Web Data Collection](https://mlsanity.com/paper/2505.15859/) (2025)
- [Symbiotic Cooperation for Web Agents: Harnessing Complementary Strengths of Large and Small LLMs](https://mlsanity.com/paper/2502.07942/) (2025)
- [R2D2: Remembering, Replaying and Dynamic Decision Making with a Reflective Agentic Memory](https://mlsanity.com/paper/2501.12485/) (2025)
- [SOP-Agent: Empower General Purpose AI Agent with Domain-Specific SOPs](https://mlsanity.com/paper/2501.09316/) (2025)
- [Systematic Categorization, Construction and Evaluation of New Attacks against Multi-modal Mobile GUI Agents](https://mlsanity.com/paper/2407.09295/) (2024)
- [Genesis: Evolving Attack Strategies for LLM Web Agent Red-Teaming](https://mlsanity.com/paper/2510.18314/) (2025)
- [Adaptive Reinforcement Learning Planning: Harnessing Large Language Models for Complex Information Extraction](https://mlsanity.com/paper/2406.11455/) (2024)
- [XPath Agent: An Efficient XPath Programming Agent Based on LLM for Web Crawler](https://mlsanity.com/paper/2502.15688/) (2024)
- [Web Page Classification using LLMs for Crawling Support](https://mlsanity.com/paper/2505.06972/) (2025)
- [Throttling Web Agents Using Reasoning Gates](https://mlsanity.com/paper/2509.01619/) (2025)
- [The AI Committee: A Multi-Agent Framework for Automated Validation and Remediation of Web-Sourced Data](https://mlsanity.com/paper/2512.21481/) (2025)
- [Beyond Text: Characterizing Domain Expert Needs in Document Research](https://mlsanity.com/paper/2504.12495/) (2025)
- [Evaluation of LLM-based Strategies for the Extraction of Food Product Information from Online Shops](https://mlsanity.com/paper/2506.21585/) (2025)
- [LiveWeb-IE: A Benchmark For Online Web Information Extraction](https://mlsanity.com/paper/2603.13773/) (2026)
- [A Multi-Agent System for Building-Age Cohort Mapping to Support Urban Energy Planning](https://mlsanity.com/paper/2603.17626/) (2026)
- [Webscraper: Leverage Multimodal Large Language Models for Index-Content Web Scraping](https://mlsanity.com/paper/2603.29161/) (2026)
- [Co-Scraper: query-aware DOM Pruning and Reusable Scraper Synthesis for Lightweight Web Data Extraction](https://mlsanity.com/paper/2606.14821/) (2026)
- [Making Failure Safe: A Constrained, Verifiable Agent Framework for Open-Web Data Collection](https://mlsanity.com/paper/2607.00035/) (2026)
- [Web Price Extraction: State of the Art and an Adaptive Browserless Implementation](https://mlsanity.com/paper/2609.01030/) (2026)
- [CCQ: A Multi-State Child Care Quality Dataset to Support AI for Children's Health Research](https://mlsanity.com/paper/2610.05863/) (2026)

## Abstract (arXiv)

Web scraping is a powerful technique that extracts data from websites, enabling automated data collection, enhancing data analysis capabilities, and minimizing manual data entry efforts. Existing methods, wrappers-based methods suffer from limited adaptability and scalability when faced with a new website, while language agents, empowered by large language models (LLMs), exhibit poor reusability in diverse web environments. In this work, we introduce the paradigm of generating web scrapers with LLMs and propose AutoScraper, a two-stage framework that can handle diverse and changing web environments more efficiently. AutoScraper leverages the hierarchical structure of HTML and similarity across different web pages for generating web scrapers. Besides, we propose a new executability metric for better measuring the performance of web scraper generation tasks. We conduct comprehensive experiments with multiple LLMs and demonstrate the effectiveness of our framework. Resources of this paper can be found at urlhttps://github.com/EZ-hwh/AutoScraper