# Synthetic Sandbox for Training Machine Learning Engineering Agents

arXiv: https://arxiv.org/abs/2604.04872 · HTML: https://mlsanity.com/paper/2604.04872/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 29 of its 36 references that are in the map.
- First posted on arXiv 6 Apr 2026.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 29 of 29)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023)
- [SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training](https://mlsanity.com/paper/2501.17161/) (2025)
- [MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering](https://mlsanity.com/paper/2410.07095/) (2024)
- [AIDE: AI-Driven Exploration in the Space of Code](https://mlsanity.com/paper/2502.13138/) (2025)
- [The Landscape of Agentic Reinforcement Learning for LLMs: A Survey](https://mlsanity.com/paper/2509.02547/) (2025)
- [WebDancer: Towards Autonomous Information Seeking Agency](https://mlsanity.com/paper/2505.22648/) (2025)
- [ReSum: Unlocking Long-Horizon Search Intelligence via Context Summarization](https://mlsanity.com/paper/2509.13313/) (2025)
- [Agentic Reasoning and Tool Integration for LLMs via Reinforcement Learning](https://mlsanity.com/paper/2505.01441/) (2025)
- [AI Research Agents for Machine Learning: Search, Exploration, and Generalization in MLE-bench](https://mlsanity.com/paper/2507.02554/) (2025)
- [ML-Master: Towards AI-for-AI via Integration of Exploration and Reasoning](https://mlsanity.com/paper/2506.16499/) (2025)
- [MLE-STAR: Machine Learning Engineering Agent via Search and Targeted Refinement](https://mlsanity.com/paper/2506.15692/) (2025)
- [Scaling Agent Learning via Experience Synthesis](https://mlsanity.com/paper/2511.03773/) (2025)
- [MLE-Dojo: Interactive Environments for Empowering LLM Agents in Machine Learning Engineering](https://mlsanity.com/paper/2505.07782/) (2025)
- [R&D-Agent: An LLM-Agent Framework Towards Autonomous Data Science](https://mlsanity.com/paper/2505.14738/) (2025)
- [Toward Ultra-Long-Horizon Agentic Science: Cognitive Accumulation for Machine Learning Engineering](https://mlsanity.com/paper/2601.10402/) (2026)
- [ML-Agent: Reinforcing LLM Agents for Autonomous Machine Learning Engineering](https://mlsanity.com/paper/2505.23723/) (2025)
- [The FM Agent](https://mlsanity.com/paper/2510.26144/) (2025)
- [Reinforcement Learning for Machine Learning Engineering Agents](https://mlsanity.com/paper/2509.01684/) (2025)
- [Let it Calm: Exploratory Annealed Decoding for Verifiable Reinforcement Learning](https://mlsanity.com/paper/2510.05251/) (2025)
- [MLE-Smith: Scaling MLE Tasks with Automated Multi-Agent Pipeline](https://mlsanity.com/paper/2510.07307/) (2025)
- [DISCO Balances the Scales: Adaptive Domain- and Difficulty-Aware Reinforcement Learning on Imbalanced Data](https://mlsanity.com/paper/2505.15074/) (2025)
- [AceGRPO: Adaptive Curriculum Enhanced Group Relative Policy Optimization for Autonomous Machine Learning Engineering](https://mlsanity.com/paper/2602.07906/) (2026)

## Built on by (most cited first, 11 of 11)

- [Qwen-AgentWorld: Language World Models for General Agents](https://mlsanity.com/paper/2606.24597/) (2026)
- [Who&When Pro: Can LLMs Really Attribute Failures in AI Agents?](https://mlsanity.com/paper/2607.09996/) (2026)
- [SAGE-OPD: Selective Agent-Guided Intervention for Multi-Turn On-Policy Distillation](https://mlsanity.com/paper/2606.19659/) (2026)
- [SPADE: Self-Play in Adaptive Synthetic Executable Environments](https://mlsanity.com/paper/2608.19197/) (2026)
- [Frontis-MA1: Training an AI4AI Model towards Recursive Self-Improvement in Machine Learning Engineering](https://mlsanity.com/paper/2607.28568/) (2026)
- [Scaling Automatic Research Agents via World Models](https://mlsanity.com/paper/2608.12564/) (2026)
- [Scaling Scientific Discovery Environments for Turn-Level Agentic RL](https://mlsanity.com/paper/2607.28990/) (2026)
- [Learning Generalizable Behaviors for Terminal Agents](https://mlsanity.com/paper/2608.22631/) (2026)
- [MEGA: Self-Evolving Agent Optimization Infrastructure via Wisdom Graph](https://mlsanity.com/paper/2608.10504/) (2026)
- [VTOS: Learning to Orchestrate Vision Tools by Co-Searching Solutions and Observers](https://mlsanity.com/paper/2606.20728/) (2026)
- [How Much of a Harness Does a Strong Agent Need for Autonomous ML Engineering?](https://mlsanity.com/paper/2609.40303/) (2026)

## Abstract (arXiv)

As large language model agents advance beyond software engineering (SWE) tasks toward machine learning engineering (MLE), verifying agent behavior becomes orders of magnitude more expensive: while SWE tasks can be verified via fast-executing unit tests, MLE verification requires running full ML pipelines -- data preprocessing, model training, and metric evaluation -- on large datasets at each rollout step, rendering trajectory-wise on-policy reinforcement learning (RL) prohibitively slow. Existing approaches retreat to supervised fine-tuning (SFT) or offline proxy rewards, sacrificing the exploration and generalization benefits of on-policy RL. We observe that sandbox data size is the primary source of this bottleneck. Based on this insight, we introduce SandMLE, a multi-agent framework that generates diverse, verifiable synthetic MLE environments from a small number of seed tasks, preserving the structural and technical complexity of real-world problems while constraining datasets to micro-scale (each task is paired with only 50-200 training samples). Through extensive experiments, we show that SandMLE reduces execution time by over 13 times, enabling large-scale, on-policy trajectory-wise RL for the first time in the MLE domain. On MLE-bench-lite, SandMLE yields significant gains over SFT baselines across Qwen3-8B, 14B, and 30B-A3B, with relative medal rate improvements ranging from 20.3% to 66.9%. Furthermore, the trained policy generalizes across unseen agentic scaffolds, achieving up to 32.4% better HumanRank score on MLE-Dojo.