# ARE: Scaling Up Agent Environments and Evaluations

arXiv: https://arxiv.org/abs/2509.17158 · HTML: https://mlsanity.com/paper/2509.17158/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use.
- Builds on 24 of its 38 references that are in the map.
- First posted on arXiv 21 Sep 2025.

## Placed in

- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)

## Builds on (most cited first, 24 of 24)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains](https://mlsanity.com/paper/2406.12045/) (2024)
- [GAIA: a benchmark for General AI Assistants](https://mlsanity.com/paper/2311.12983/) (2023)
- [Tulu 3: Pushing Frontiers in Open Language Model Post-Training](https://mlsanity.com/paper/2411.15124/) (2024)
- [BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents](https://mlsanity.com/paper/2504.12516/) (2025)
- [Learning to Communicate with Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1605.06676/) (2016)
- [Scaling Synthetic Data Creation with 1,000,000,000 Personas](https://mlsanity.com/paper/2406.20094/) (2024)
- [AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents](https://mlsanity.com/paper/2407.18901/) (2024)
- [On the Utility of Learning about Humans for Human-AI Coordination](https://mlsanity.com/paper/1910.05789/) (2019)
- [ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilities](https://mlsanity.com/paper/2408.04682/) (2024)
- [SWE-smith: Scaling Data for Software Engineering Agents](https://mlsanity.com/paper/2504.21798/) (2025)
- [Small Language Models are the Future of Agentic AI](https://mlsanity.com/paper/2506.02153/) (2025)
- [MultiAgentBench: Evaluating the Collaboration and Competition of LLM agents](https://mlsanity.com/paper/2503.01935/) (2025)
- [WebAgent-R1: Training Web Agents via End-to-End Multi-Turn Reinforcement Learning](https://mlsanity.com/paper/2505.16421/) (2025)
- [RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning](https://mlsanity.com/paper/2410.02089/) (2024)
- [Generative agent-based modeling with actions grounded in physical, social, or digital space using Concordia](https://mlsanity.com/paper/2312.03664/) (2023)
- [MCP-Bench: Benchmarking Tool-Using LLM Agents with Complex Real-World Tasks via MCP Servers](https://mlsanity.com/paper/2508.20453/) (2025)
- [Rule Based Rewards for Language Model Safety](https://mlsanity.com/paper/2411.01111/) (2024)
- [Vending-Bench: A Benchmark for Long-Term Coherence of Autonomous Agents](https://mlsanity.com/paper/2502.15840/) (2025)
- [Interactive Debugging and Steering of Multi-Agent AI Systems](https://mlsanity.com/paper/2503.02068/) (2025)
- [MCP-RADAR: A Multi-Dimensional Benchmark for Evaluating Tool Use Capabilities in Large Language Models](https://mlsanity.com/paper/2505.16700/) (2025)
- [Interactive Reasoning: Visualizing and Controlling Chain-of-Thought Reasoning in Large Language Models](https://mlsanity.com/paper/2506.23678/) (2025)

## Built on by (most cited first, 30 of 30)

- [Agent-World: Scaling Real-World Environment Synthesis for Evolving General Agent Intelligence](https://mlsanity.com/paper/2604.18292/) (2026)
- [APEX-Agents](https://mlsanity.com/paper/2601.14242/) (2026)
- [SkillCraft: Can LLM Agents Learn to Use Tools Skillfully?](https://mlsanity.com/paper/2603.00718/) (2026)
- [Immersion in the GitHub Universe: Scaling Coding Agents to Mastery](https://mlsanity.com/paper/2602.09892/) (2026)
- [Nex-N1: Agentic Models Trained via a Unified Ecosystem for Large-Scale Environment Construction](https://mlsanity.com/paper/2512.04987/) (2025)
- [ASTRA-bench: Evaluating Tool-Use Agent Reasoning and Action Planning with Personal User Context](https://mlsanity.com/paper/2603.01357/) (2026)
- [DIVE: Scaling Diversity in Agentic Task Synthesis for Generalizable Tool Use](https://mlsanity.com/paper/2603.11076/) (2026)
- [OpenApps: Simulating Environment Variations to Measure UI-Agent Reliability](https://mlsanity.com/paper/2511.20766/) (2025)
- [M^3-Bench: Multi-Modal, Multi-Hop, Multi-Threaded Tool-Using MLLM Agent Benchmark](https://mlsanity.com/paper/2511.17729/) (2025)
- [Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Application](https://mlsanity.com/paper/2606.12191/) (2026)
- [LongCoT: Benchmarking Long-Horizon Chain-of-Thought Reasoning](https://mlsanity.com/paper/2604.14140/) (2026)
- [Ling and Ring 2.6 Technical Report: Efficient and Instant Agentic Intelligence at Trillion-Parameter Scale](https://mlsanity.com/paper/2606.15079/) (2026)
- [Agentic Discovery of Neural Architectures: AIRA-Compose and AIRA-Design](https://mlsanity.com/paper/2605.15871/) (2026)
- [CreativityBench: Evaluating Agent Creative Reasoning via Affordance-Based Tool Repurposing](https://mlsanity.com/paper/2605.02910/) (2026)
- [Teach the Magnitude, Not the Direction: Verifier-Bounded Credit Assignment for Multi-Turn Multi-step LLM Agents](https://mlsanity.com/paper/2608.13179/) (2026)
- [Automating Benchmark Design](https://mlsanity.com/paper/2510.25039/) (2025)
- [The Agent's First Day: Benchmarking Learning, Exploration, and Scheduling in the Workplace Scenarios](https://mlsanity.com/paper/2601.08173/) (2026)
- [Scaling Small Agents Through Strategy Auctions](https://mlsanity.com/paper/2602.02751/) (2026)
- [Interactive Evaluation Requires a Design Science](https://mlsanity.com/paper/2605.17829/) (2026)
- [SentinelBench: A Benchmark for Long-Running Monitoring Agents](https://mlsanity.com/paper/2606.05342/) (2026)
- [Reliable agent engineering should integrate machine-compatible organizational principles](https://mlsanity.com/paper/2512.07665/) (2025)
- [Scalable Environments Drive Generalizable Agents](https://mlsanity.com/paper/2605.18181/) (2026)
- [The Path to Self-Evolving Clinical Systems: Scaling Medical Agents from Assistance to Autonomy](https://mlsanity.com/paper/2607.11175/) (2026)
- [RCTs for Frontier AI Governance: Methodological Challenges and Solutions for Human Uplift Studies](https://mlsanity.com/paper/2603.11001/) (2026)
- [RevengeBench: Reverse Engineering Code-Space Policies from Behavioral Experiments](https://mlsanity.com/paper/2606.26094/) (2026)
- [Deploying and Evaluating a Smart-Agriculture Agentic Engine for Full-Season Soybean Farm Operations](https://mlsanity.com/paper/2609.00106/) (2026)
- [Trust and Task Completion in the World of Consumer AI Agents](https://mlsanity.com/paper/2609.33017/) (2026)
- [ReLiveGym: Evaluating Long-Lived Agents over Weeks of Replayed Reality](https://mlsanity.com/paper/2610.00710/) (2026)
- [AgentSpy: Making AI Agent Behavior Observable](https://mlsanity.com/paper/2610.06001/) (2026)
- [Proactive Agentic Orchestration](https://mlsanity.com/paper/2610.13485/) (2026)

## Abstract (arXiv)

We introduce Meta Agents Research Environments (ARE), a research platform for scalable creation of environments, integration of synthetic or real applications, and execution of agentic orchestrations. ARE provides simple abstractions to build complex and diverse environments, each with their own rules, tools, content, and verifiers, helping to bridge the gap between model development and real-world deployment. We also propose Gaia2, a benchmark built in ARE and designed to measure general agent capabilities. Beyond search and execution, Gaia2 requires agents to handle ambiguities and noise, adapt to dynamic environments, collaborate with other agents, and operate under temporal constraints. Unlike prior benchmarks, Gaia2 runs asynchronously, surfacing new failure modes that are invisible in static settings. Our experiments show that no system dominates across the intelligence spectrum: stronger reasoning often comes at the cost of efficiency, and budget scaling curves plateau, highlighting the need for new architectures and adaptive compute strategies. Perhaps more importantly, ARE abstractions enable continuous extension of Gaia2 to other environments, empowering the community to rapidly create new benchmarks tailored to their domains. In AI's second half, progress increasingly depends on defining meaningful tasks and robust evaluations to drive frontier capabilities forward.