# AutoData: A Multi-Agent System for Open Web Data Collection

arXiv: https://arxiv.org/abs/2505.15859 · HTML: https://mlsanity.com/paper/2505.15859/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use.
- Builds on 45 of its 75 references that are in the map.
- First posted on arXiv 21 May 2025.

## Placed in

- LLM agents and tool use / [Web, search, and research agents](https://mlsanity.com/agents/web-search-and-research-agents/)

## Builds on (most cited first, 30 of 45)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023)
- [AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation](https://mlsanity.com/paper/2308.08155/) (2023)
- [ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs](https://mlsanity.com/paper/2307.16789/) (2023)
- [MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework](https://mlsanity.com/paper/2308.00352/) (2023)
- [Improving Factuality and Reasoning in Language Models through Multiagent Debate](https://mlsanity.com/paper/2305.14325/) (2023)
- [The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale](https://mlsanity.com/paper/2406.17557/) (2024)
- [ChatDev: Communicative Agents for Software Development](https://mlsanity.com/paper/2307.07924/) (2023)
- [Large Language Model based Multi-Agents: A Survey of Progress and Challenges](https://mlsanity.com/paper/2402.01680/) (2024)
- [ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate](https://mlsanity.com/paper/2308.07201/) (2023)
- [A Survey on the Memory Mechanism of Large Language Model based Agents](https://mlsanity.com/paper/2404.13501/) (2024)
- [End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF](https://mlsanity.com/paper/1603.01354/) (2016)
- [Tool Learning with Foundation Models](https://mlsanity.com/paper/2304.08354/) (2023)
- [A Survey on Deep Learning for Named Entity Recognition](https://mlsanity.com/paper/1812.09449/) (2018)
- [Self-collaboration Code Generation via ChatGPT](https://mlsanity.com/paper/2304.07590/) (2023)
- [Building Cooperative Embodied Agents Modularly with Large Language Models](https://mlsanity.com/paper/2307.02485/) (2023)
- [The Landscape of Emerging AI Agent Architectures for Reasoning, Planning, and Tool Calling: A Survey](https://mlsanity.com/paper/2404.11584/) (2024)
- [OpenWebMath: An Open Dataset of High-Quality Mathematical Web Text](https://mlsanity.com/paper/2310.06786/) (2023)
- [AgentCF: Collaborative Learning with Autonomous Language Agents for Recommender Systems](https://mlsanity.com/paper/2310.09233/) (2023)
- [FinRL: A Deep Reinforcement Learning Library for Automated Stock Trading in Quantitative Finance](https://mlsanity.com/paper/2011.09607/) (2020)
- [From LLMs to LLM-based Agents for Software Engineering: A Survey of Current, Challenges and Future](https://mlsanity.com/paper/2408.02479/) (2024)
- [Datasets for Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2402.18041/) (2024)
- [OlympicArena: Benchmarking Multi-discipline Cognitive Reasoning for Superintelligent AI](https://mlsanity.com/paper/2406.12753/) (2024)

## Built on by (most cited first, 25 of 25)

- [AgentRouter: A Knowledge-Graph-Guided LLM Router for Collaborative Multi-Agent Question Answering](https://mlsanity.com/paper/2510.05445/) (2025)
- [Why Reasoning Fails to Plan: A Planning-Centric Analysis of Long-Horizon Decision Making in LLM Agents](https://mlsanity.com/paper/2601.22311/) (2026)
- [Autodata: An agentic data scientist to create high quality synthetic data](https://mlsanity.com/paper/2606.25996/) (2026)
- [CiteAudit: You Cited It, But Did You Read It? A Benchmark for Verifying Scientific References in the LLM Era](https://mlsanity.com/paper/2602.23452/) (2026)
- [EvolveRouter: Co-Evolving Routing and Prompt for Multi-Agent Question Answering](https://mlsanity.com/paper/2604.05149/) (2026)
- [NG-Router: Graph-Supervised Multi-Agent Collaboration for Nutrition Question Answering](https://mlsanity.com/paper/2510.09854/) (2025)
- [Drift-Bench: Diagnosing Cooperative Breakdowns in LLM Agents under Input Faults via Multi-Turn Interaction](https://mlsanity.com/paper/2602.02455/) (2026)
- [Same Signal, Opposite Meaning: Direction-Informed Adaptive Learning for LLM Agents](https://mlsanity.com/paper/2605.06908/) (2026)
- [CollabEval: Enhancing LLM-as-a-Judge via Multi-Agent Collaboration](https://mlsanity.com/paper/2603.00993/) (2026)
- [Bridging Modalities, Spanning Time: Structured Memory for Ultra-Long Agentic Video Reasoning](https://mlsanity.com/paper/2605.08271/) (2026)
- [Source or It Didn't Happen: A Multi-Agent Framework for Citation Hallucination Detection](https://mlsanity.com/paper/2605.08583/) (2026)
- [SAGE: Answer-Conditioned Uncertainty Targets for Verbal Uncertainty Alignment](https://mlsanity.com/paper/2606.11512/) (2026)
- [BHyGNN+: Unsupervised Representation Learning for Heterophilic Hypergraphs](https://mlsanity.com/paper/2602.14919/) (2026)
- [MDForge: Agentic Molecular Dynamics Pipeline Design under Sparse Simulator Feedback](https://mlsanity.com/paper/2606.12916/) (2026)
- [OPBench: A Graph Benchmark to Combat the Opioid Crisis](https://mlsanity.com/paper/2602.14602/) (2026)
- [Brick-DICL: Dynamic In-Context Learning for Automated Brick Schema Classification](https://mlsanity.com/paper/2606.17637/) (2026)
- [Confidence Laundering in Agent Systems: Why Uncertainty Needs a Latent Carrier](https://mlsanity.com/paper/2606.20662/) (2026)
- [Can AI autonomously build, operate, and use the entire data stack?](https://mlsanity.com/paper/2512.07926/) (2025)
- [SupraBench: A Benchmark for Supramolecular Chemistry](https://mlsanity.com/paper/2606.13477/) (2026)
- [Making Failure Safe: A Constrained, Verifiable Agent Framework for Open-Web Data Collection](https://mlsanity.com/paper/2607.00035/) (2026)
- [HoosierHelp: Benchmarking LLM Agents for Social Service Navigation](https://mlsanity.com/paper/2608.09946/) (2026)
- [JudgePanel: A Compact Judge with Panel Deliberation via Adaptive Multi-Reward Reinforcement Learning](https://mlsanity.com/paper/2608.29168/) (2026)
- [Data Scout: Targeted Web Crawling for Domain-Specific Pretraining Corpora](https://mlsanity.com/paper/2609.05766/) (2026)
- [Towards Reliable AI Data Scientists: Data Agents with Workflow Harnesses](https://mlsanity.com/paper/2609.35255/) (2026)
- [Concealing LLM-Based Multi-Agent Topology via Phantom Structure Injection](https://mlsanity.com/paper/2609.37567/) (2026)

## Abstract (arXiv)

The exponential growth of data-driven systems and AI technologies has intensified the demand for high-quality web-sourced datasets. While existing datasets have proven valuable, conventional web data collection approaches face significant limitations in terms of human effort and scalability. Current data-collecting solutions fall into two categories: wrapper-based methods that struggle with adaptability and reproducibility, and large language model (LLM)-based approaches that incur substantial computational and financial costs. To address these challenges, we propose AutoData, a novel multi-agent system for Automated web Data collection, that requires minimal human intervention, i.e., only necessitating a natural language instruction specifying the desired dataset. In addition, AutoData is designed with a robust multi-agent architecture, featuring a novel oriented message hypergraph coordinated by a central task manager, to efficiently organize agents across research and development squads. Besides, we introduce a novel hypergraph cache system to advance the multi-agent collaboration process that enables efficient automated data collection and mitigates the token cost issues prevalent in existing LLM-based systems. Moreover, we introduce Instruct2DS, a new benchmark dataset supporting live data collection from web sources across three domains: academic, finance, and sports. Comprehensive evaluations over Instruct2DS and three existing benchmark datasets demonstrate AutoData's superior performance compared to baseline methods. Case studies on challenging tasks such as picture book collection and paper extraction from surveys further validate its applicability. Our source code and dataset are available at https://github.com/GraphResearcher/AutoData.