# Towards Adaptive ML Benchmarks: Web-Agent-Driven Construction, Domain Expansion, and Metric Optimization

arXiv: https://arxiv.org/abs/2509.09321 · HTML: https://mlsanity.com/paper/2509.09321/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use.
- Builds on 11 of its 14 references that are in the map.
- First posted on arXiv 11 Sep 2025.

## Placed in

- LLM agents and tool use / [Web, search, and research agents](https://mlsanity.com/agents/web-search-and-research-agents/)

## Builds on (most cited first, 11 of 11)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [OpenHands: An Open Platform for AI Software Developers as Generalist Agents](https://mlsanity.com/paper/2407.16741/) (2024)
- [DeepSeek-V3 Technical Report](https://mlsanity.com/paper/2412.19437/) (2024)
- [MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering](https://mlsanity.com/paper/2410.07095/) (2024)
- [MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation](https://mlsanity.com/paper/2310.03302/) (2023)
- [Model Context Protocol (MCP): Landscape, Security Threats, and Future Research Directions](https://mlsanity.com/paper/2503.23278/) (2025)
- [AIDE: AI-Driven Exploration in the Space of Code](https://mlsanity.com/paper/2502.13138/) (2025)
- [ML-Master: Towards AI-for-AI via Integration of Exploration and Reasoning](https://mlsanity.com/paper/2506.16499/) (2025)
- [ML-Agent: Reinforcing LLM Agents for Autonomous Machine Learning Engineering](https://mlsanity.com/paper/2505.23723/) (2025)
- [Empowering Many, Biasing a Few: Generalist Credit Scoring through Large Language Models](https://mlsanity.com/paper/2310.00566/) (2023)
- [AutoMind: Adaptive Knowledgeable Agent for Automated Data Science](https://mlsanity.com/paper/2506.10974/) (2025)

## Built on by (most cited first, 5 of 5)

- [From Question Answering to Task Completion: A Survey on Agent System and Harness Design](https://mlsanity.com/paper/2606.20683/) (2026)
- [What Makes AI Research Replicable? Executable Knowledge Graphs as Scientific Knowledge Representations](https://mlsanity.com/paper/2510.17795/) (2025)
- [Can Large Language Models Derive New Knowledge? A Dynamic Benchmark for Biological Knowledge Discovery](https://mlsanity.com/paper/2603.03322/) (2026)
- [ParEvalLayer: When Partial LLM-Agent Evaluations Support a Decision](https://mlsanity.com/paper/2608.02444/) (2026)
- [A Framework for Assessing AI Agent Decisions and Outcomes in AutoML Pipelines](https://mlsanity.com/paper/2602.22442/) (2026)

## Abstract (arXiv)

Recent advances in large language models (LLMs) have enabled the emergence of general-purpose agents for automating end-to-end machine learning (ML) workflows, including data analysis, feature engineering, model training, and competition solving. However, existing benchmarks remain limited in task coverage, domain diversity, difficulty modeling, and evaluation rigor, failing to capture the full capabilities of such agents in realistic settings. We present TAM Bench, a diverse, realistic, and structured benchmark for evaluating LLM-based agents on end-to-end ML tasks. TAM Bench features three key innovations: (1) A browser automation and LLM-based task acquisition system that automatically collects and structures ML challenges from platforms such as Kaggle, AIcrowd, and Biendata, spanning multiple task types and data modalities (e.g., tabular, text, image, graph, audio); (2) A leaderboard-driven difficulty modeling mechanism that estimates task complexity using participant counts and score dispersion, enabling scalable and objective task calibration; (3) A multi-dimensional evaluation framework incorporating performance, format compliance, constraint adherence, and task generalization. Based on 150 curated AutoML tasks, we construct three benchmark subsets of different sizes -- Lite, Medium, and Full -- designed for varying evaluation scenarios. The Lite version, with 18 tasks and balanced coverage across modalities and difficulty levels, serves as a practical testbed for daily benchmarking and comparative studies.