# Making Failure Safe: A Constrained, Verifiable Agent Framework for Open-Web Data Collection

arXiv: https://arxiv.org/abs/2607.00035 · HTML: https://mlsanity.com/paper/2607.00035/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use.
- Builds on 13 of its 27 references that are in the map.
- First posted on arXiv 25 Jun 2026.

## Placed in

- LLM agents and tool use / [Web, search, and research agents](https://mlsanity.com/agents/web-search-and-research-agents/)

## Builds on (most cited first, 13 of 13)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023)
- [CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis](https://mlsanity.com/paper/2203.13474/) (2022)
- [WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models](https://mlsanity.com/paper/2401.13919/) (2024)
- [AutoData: A Multi-Agent System for Open Web Data Collection](https://mlsanity.com/paper/2505.15859/) (2025)
- [AutoScraper: A Progressive Understanding Web Agent for Web Scraper Generation](https://mlsanity.com/paper/2404.12753/) (2024)
- [Fight Fire with Fire: How Much Can We Trust ChatGPT on Source Code-Related Tasks?](https://mlsanity.com/paper/2405.12641/) (2024)
- [PARSE: LLM Driven Schema Optimization for Reliable Entity Extraction](https://mlsanity.com/paper/2510.08623/) (2025)
- [The AI Committee: A Multi-Agent Framework for Automated Validation and Remediation of Web-Sourced Data](https://mlsanity.com/paper/2512.21481/) (2025)

## Abstract (arXiv)

LLMs and agents can generate web scrapers from natural-language requirements, but direct generation remains unreliable because of dependency errors, broken selectors, schema mismatches, and heterogeneous page structures. We propose a constrained, verifiable agent framework that shifts LLM output from free-form code to typed JSON collector configurations, combining a six-type collector taxonomy, template and utility-function constraints, static Airflow DAG execution, rule-based quality checking, and structured feedback correction. Experiments on 138 tasks show that the taxonomy supports description-based requirement typing, while confirming that stable instantiation requires completing source, field, and execution constraints beyond the initial description. On 80 independently source-verified tasks, the framework runs with zero execution-stage LLM tokens and the lowest average wall-clock time, trading moderate one-shot quality for a reusable, deterministic, and verifiable execution path suited to repeated scheduled collection. These results position the framework as a reusable, low-cost, and verifiable execution path for repeated open-web data collection.