# XPath Agent: An Efficient XPath Programming Agent Based on LLM for Web Crawler

arXiv: https://arxiv.org/abs/2502.15688 · HTML: https://mlsanity.com/paper/2502.15688/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 24 references that are in the map.
- First posted on arXiv 17 Dec 2024.

## Builds on (most cited first, 7 of 7)

- [ChatIE: Zero-Shot Information Extraction via Chatting with ChatGPT](https://mlsanity.com/paper/2302.10205/) (2023)
- [MarkupLM: Pre-training of Text and Markup Language for Visually-rich Document Understanding](https://mlsanity.com/paper/2110.08518/) (2021)
- [WebFormer: The Web-page Transformer for Structure Information Extraction](https://mlsanity.com/paper/2202.00217/) (2022)
- [FreeDOM: A Transferable Neural Architecture for Structured Information Extraction on Web Documents](https://mlsanity.com/paper/2010.10755/) (2020)
- [AutoScraper: A Progressive Understanding Web Agent for Web Scraper Generation](https://mlsanity.com/paper/2404.12753/) (2024)
- [Simplified DOM Trees for Transferable Attribute Extraction from the Web](https://mlsanity.com/paper/2101.02415/) (2021)
- [Leveraging Large Language Models for Web Scraping](https://mlsanity.com/paper/2406.08246/) (2024)

## Built on by (most cited first, 2 of 2)

- [LLM/Agent-as-Data-Analyst: A Survey](https://mlsanity.com/paper/2509.23988/) (2025)
- [Webscraper: Leverage Multimodal Large Language Models for Index-Content Web Scraping](https://mlsanity.com/paper/2603.29161/) (2026)

## Abstract (arXiv)

We present XPath Agent, a production-ready XPath programming agent specifically designed for web crawling and web GUI testing. A key feature of XPath Agent is its ability to automatically generate XPath queries from a set of sampled web pages using a single natural language query. To demonstrate its effectiveness, we benchmark XPath Agent against a state-of-the-art XPath programming agent across a range of web crawling tasks. Our results show that XPath Agent achieves comparable performance metrics while significantly reducing token usage and improving clock-time efficiency. The well-designed two-stage pipeline allows for seamless integration into existing web crawling or web GUI testing workflows, thereby saving time and effort in manual XPath query development. The source code for XPath Agent is available at https://github.com/eavae/feilian.