# Detecting Pipeline Failures through Fine-Grained Analysis of Web Agents

arXiv: https://arxiv.org/abs/2509.14382 · HTML: https://mlsanity.com/paper/2509.14382/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use.
- Builds on 18 of its 28 references that are in the map.
- First posted on arXiv 17 Sep 2025.

## Placed in

- LLM agents and tool use / [Web, search, and research agents](https://mlsanity.com/agents/web-search-and-research-agents/)

## Builds on (most cited first, 18 of 18)

- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [Mind2Web: Towards a Generalist Agent for the Web](https://mlsanity.com/paper/2306.06070/) (2023)
- [GPT-4V(ision) is a Generalist Web Agent, if Grounded](https://mlsanity.com/paper/2401.01614/) (2024)
- [Language Models can Solve Computer Tasks](https://mlsanity.com/paper/2303.17491/) (2023)
- [A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis](https://mlsanity.com/paper/2307.12856/) (2023)
- [Multimodal Web Navigation with Instruction-Finetuned Foundation Models](https://mlsanity.com/paper/2305.11854/) (2023)
- [Efficient Few-Shot Learning Without Prompts](https://mlsanity.com/paper/2209.11055/) (2022)
- [AutoWebGLM: A Large Language Model-based Web Navigating Agent](https://mlsanity.com/paper/2404.03648/) (2024)
- [WebCanvas: Benchmarking Web Agents in Online Environments](https://mlsanity.com/paper/2406.12373/) (2024)
- [TaskWeaver: A Code-First Agent Framework](https://mlsanity.com/paper/2311.17541/) (2023)
- [Agent Lumos: Unified and Modular Training for Open-Source Language Agents](https://mlsanity.com/paper/2311.05657/) (2023)
- [SteP: Stacked LLM Policies for Web Actions](https://mlsanity.com/paper/2310.03720/) (2023)
- [Grounding Open-Domain Instructions to Automate Web Support Tasks](https://mlsanity.com/paper/2103.16057/) (2021)
- [Hierarchical Prompting Assists Large Language Model on Web Navigation](https://mlsanity.com/paper/2305.14257/) (2023)
- [FLIN: A Flexible Natural Language Interface for Web Navigation](https://mlsanity.com/paper/2010.12844/) (2020)
- [LiteWebAgent: The Open-Source Suite for VLM-Based Web-Agent Applications](https://mlsanity.com/paper/2503.02950/) (2025)
- [NaviQAte: Functionality-Guided Web Application Navigation](https://mlsanity.com/paper/2409.10741/) (2024)
- [Towards a Realistic Long-Term Benchmark for Open-Web Research Agents](https://mlsanity.com/paper/2409.14913/) (2024)

## Abstract (arXiv)

Web agents powered by large language models (LLMs) can autonomously perform complex, multistep tasks in dynamic web environments. However, current evaluations mostly focus on the overall success while overlooking intermediate errors. This limits insight into failure modes and hinders systematic improvement. This work analyzes existing benchmarks and highlights the lack of fine-grained diagnostic tools. To address this gap, we propose a modular evaluation framework that decomposes agent pipelines into interpretable stages for detailed error analysis. Using the SeeAct framework and the Mind2Web dataset as a case study, we show how this approach reveals actionable weaknesses missed by standard metrics - paving the way for more robust and generalizable web agents.