# TimeSeriesGym: A Scalable Benchmark for (Time Series) Machine Learning Engineering Agents arXiv: https://arxiv.org/abs/2505.13291 ยท HTML: https://mlsanity.com/paper/2505.13291/ ## Key facts - Cited by 15 papers in the map (data through 14 Oct 2026). - Placed in Financial trading and forecasting agents, one of 16 directions in LLM agents and tool use. - Builds on 16 of its 23 references that are in the map. - First posted on arXiv 19 May 2025. ## Placed in - LLM agents and tool use / [Financial trading and forecasting agents](https://mlsanity.com/agents/financial-trading-and-forecasting-agents/) ## Builds on (most cited first, 16 of 16) - [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023) - [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024) - [GAIA: a benchmark for General AI Assistants](https://mlsanity.com/paper/2311.12983/) (2023) - [OpenHands: An Open Platform for AI Software Developers as Generalist Agents](https://mlsanity.com/paper/2407.16741/) (2024) - [MOMENT: A Family of Open Time-series Foundation Models](https://mlsanity.com/paper/2402.03885/) (2024) - [MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering](https://mlsanity.com/paper/2410.07095/) (2024) - [MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation](https://mlsanity.com/paper/2310.03302/) (2023) - [AIDE: AI-Driven Exploration in the Space of Code](https://mlsanity.com/paper/2502.13138/) (2025) - [RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts](https://mlsanity.com/paper/2411.15114/) (2024) - [DSBench: How Far Are Data Science Agents from Becoming Data Science Experts?](https://mlsanity.com/paper/2409.07703/) (2024) - [ML-Bench: Evaluating Large Language Models and Agents for Machine Learning Tasks on Repository-Level Code](https://mlsanity.com/paper/2311.09835/) (2023) - [SUPER: Evaluating Agents on Setting Up and Executing Tasks from Research Repositories](https://mlsanity.com/paper/2409.07440/) (2024) - [TimeSeriesExam: A time series understanding exam](https://mlsanity.com/paper/2410.14752/) (2024) - [Automated Evaluation of Retrieval-Augmented Language Models with Task-Specific Exam Generation](https://mlsanity.com/paper/2405.13622/) (2024) - [ML-Dev-Bench: Comparative Analysis of AI Agents on ML development workflows](https://mlsanity.com/paper/2502.00964/) (2025) - [AQuA: A Benchmarking Tool for Label Quality Assessment](https://mlsanity.com/paper/2306.09467/) (2023) ## Built on by (most cited first, 15 of 15) - [Large Language Model-based Data Science Agent: A Survey](https://mlsanity.com/paper/2508.02744/) (2025) - [A Survey of Reasoning and Agentic Systems in Time Series with Large Language Models](https://mlsanity.com/paper/2509.11575/) (2025) - [TimeCopilot](https://mlsanity.com/paper/2509.00616/) (2025) - [TimeSeriesExamAgent: Creating Time Series Reasoning Benchmarks at Scale](https://mlsanity.com/paper/2604.10291/) (2026) - [AXIS: Explainable Time Series Anomaly Detection with Large Language Models](https://mlsanity.com/paper/2509.24378/) (2025) - [Position: Universal Time Series Foundation Models Rest on a Category Error](https://mlsanity.com/paper/2602.05287/) (2026) - [AION: Next-Generation Tasks and Practical Harness for Time Series](https://mlsanity.com/paper/2605.25045/) (2026) - [OpenFinGym: A Verifiable Multi-Task Gym Environment for Evaluating Quant Agents](https://mlsanity.com/paper/2606.26350/) (2026) - [LLM Agents for Time-Series: A Survey](https://mlsanity.com/paper/2608.26226/) (2026) - [TraceBench: Controlled Evaluation of LLM Agents for Time-Series Root-Cause Attribution](https://mlsanity.com/paper/2608.27182/) (2026) - [MILO: Automated Harness Discovery via Orchestrated Multi-Agent Evolution](https://mlsanity.com/paper/2609.38349/) (2026) - [Business Utility of Large Language Models as Exploratory Data Analysis Agents](https://mlsanity.com/paper/2606.00051/) (2026) - [Predicting Pseudo-nitzschia harmful algal blooms along the Portuguese Coast using satellite-derived predictors](https://mlsanity.com/paper/2607.07834/) (2026) - [TimeSage-EV: A Live Benchmark for Agentic Time Series Analysis in Evolving Environments](https://mlsanity.com/paper/2608.14270/) (2026) - [Forecast Workflow Bench: Evaluating Language-Model Decisions with Budgeted Forecast Tools](https://mlsanity.com/paper/2609.27385/) (2026) ## Abstract (arXiv) We introduce TimeSeriesGym, a scalable benchmarking framework for evaluating Artificial Intelligence (AI) agents on time series machine learning engineering challenges. Existing benchmarks lack scalability, focus narrowly on model building in well-defined settings, and evaluate only a limited set of research artifacts (e.g., CSV submission files). To make AI agent benchmarking more relevant to the practice of machine learning engineering, our framework scales along two critical dimensions. First, recognizing that effective ML engineering requires a range of diverse skills, TimeSeriesGym incorporates challenges from diverse sources spanning multiple domains and tasks. We design challenges to evaluate both isolated capabilities (including data handling, understanding research repositories, and code translation) and their combinations, and rather than addressing each challenge independently, we develop tools that support designing multiple challenges at scale. Second, we implement evaluation mechanisms for multiple research artifacts, including submission files, code, and models, using both precise numeric measures and more flexible LLM-based evaluation approaches. This dual strategy balances objective assessment with contextual judgment. Although our initial focus is on time series applications, our framework can be readily extended to other data modalities, broadly enhancing the comprehensiveness and practical utility of agentic AI evaluation. We open-source our benchmarking framework to facilitate future research on the ML engineering capabilities of AI agents.