# Gaia2: Benchmarking LLM Agents on Dynamic and Asynchronous Environments

arXiv: https://arxiv.org/abs/2602.11964 · HTML: https://mlsanity.com/paper/2602.11964/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use.
- Builds on 20 of its 30 references that are in the map.
- First posted on arXiv 12 Feb 2026.

## Placed in

- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)

## Builds on (most cited first, 20 of 20)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains](https://mlsanity.com/paper/2406.12045/) (2024)
- [WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents](https://mlsanity.com/paper/2207.01206/) (2022)
- [ALFWorld: Aligning Text and Embodied Environments for Interactive Learning](https://mlsanity.com/paper/2010.03768/) (2020)
- [GAIA: a benchmark for General AI Assistants](https://mlsanity.com/paper/2311.12983/) (2023)
- [BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents](https://mlsanity.com/paper/2504.12516/) (2025)
- [τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environment](https://mlsanity.com/paper/2506.07982/) (2025)
- [Scaling Synthetic Data Creation with 1,000,000,000 Personas](https://mlsanity.com/paper/2406.20094/) (2024)
- [AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents](https://mlsanity.com/paper/2407.18901/) (2024)
- [Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains](https://mlsanity.com/paper/2507.17746/) (2025)
- [WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?](https://mlsanity.com/paper/2403.07718/) (2024)
- [PaperBench: Evaluating AI's Ability to Replicate AI Research](https://mlsanity.com/paper/2504.01848/) (2025)
- [ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilities](https://mlsanity.com/paper/2408.04682/) (2024)
- [WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild](https://mlsanity.com/paper/2406.04770/) (2024)
- [MCP-Bench: Benchmarking Tool-Using LLM Agents with Complex Real-World Tasks via MCP Servers](https://mlsanity.com/paper/2508.20453/) (2025)
- [Vending-Bench: A Benchmark for Long-Term Coherence of Autonomous Agents](https://mlsanity.com/paper/2502.15840/) (2025)
- [Interactive Debugging and Steering of Multi-Agent AI Systems](https://mlsanity.com/paper/2503.02068/) (2025)
- [MCP-RADAR: A Multi-Dimensional Benchmark for Evaluating Tool Use Capabilities in Large Language Models](https://mlsanity.com/paper/2505.16700/) (2025)
- [Interactive Reasoning: Visualizing and Controlling Chain-of-Thought Reasoning in Large Language Models](https://mlsanity.com/paper/2506.23678/) (2025)

## Built on by (most cited first, 30 of 33)

- [Evolving Agents in the Dark: Retrospective Harness Optimization via Self-Preference](https://mlsanity.com/paper/2606.05922/) (2026)
- [AutoSaddler: Automatic Harness Optimization with Durable Updates from Agent Execution Traces](https://mlsanity.com/paper/2608.23041/) (2026)
- [EvoArena: Tracking Memory Evolution for Robust LLM Agents in Dynamic Environments](https://mlsanity.com/paper/2606.13681/) (2026)
- [CoffeeBench: Benchmarking Long-Horizon LLM Agents in Heterogeneous Multi-Agent Economies](https://mlsanity.com/paper/2606.16613/) (2026)
- [FutureSim: Replaying World Events to Evaluate Adaptive Agents](https://mlsanity.com/paper/2605.15188/) (2026)
- [Agentic-MME: What Agentic Capability Really Brings to Multimodal Intelligence?](https://mlsanity.com/paper/2604.03016/) (2026)
- [STT-Arena: A More Realistic Environment for Tool-Using with Spatio-Temporal Dynamics](https://mlsanity.com/paper/2605.18548/) (2026)
- [GTA-2: Benchmarking General Tool Agents from Atomic Tool-Use to Open-Ended Workflows](https://mlsanity.com/paper/2604.15715/) (2026)
- [SentinelBench: A Benchmark for Long-Running Monitoring Agents](https://mlsanity.com/paper/2606.05342/) (2026)
- [MM-ToolSandBox: A Unified Framework for Evaluating Visual Tool-Calling Agents](https://mlsanity.com/paper/2607.11818/) (2026)
- [CUBE: A Standard for Unifying Agent Benchmarks](https://mlsanity.com/paper/2603.15798/) (2026)
- [OmnilingualGAIA2: Evaluating the Multilingual Gap in Frontier AI Agents](https://mlsanity.com/paper/2608.08775/) (2026)
- [Monitoring and Discovering Reward Hacking with Internal Representations during LLM Evaluations](https://mlsanity.com/paper/2609.19101/) (2026)
- [DeliveryGym: An RL Environment for Long-Horizon Embodied Agent Planning with Adaptive Curriculum](https://mlsanity.com/paper/2609.19801/) (2026)
- [AlphaEval: Evaluating Agents in Production](https://mlsanity.com/paper/2604.12162/) (2026)
- [Mesh Memory Protocol: Semantic Infrastructure for Multi-Agent LLM Systems](https://mlsanity.com/paper/2604.19540/) (2026)
- [ORAgentBench: Can LLM Agents Solve Challenging Operations Research Tasks End to End?](https://mlsanity.com/paper/2606.19787/) (2026)
- [OmniaBench: Benchmarking General AI Agents Across Diverse Scenarios](https://mlsanity.com/paper/2607.14989/) (2026)
- [A Two-Tier Perspective on Inference-Time Parallelism in Multi-Agent LLM Systems](https://mlsanity.com/paper/2608.05791/) (2026)
- [AgentOmnia: Scaling Agentic Models for Full-Scenario Applications](https://mlsanity.com/paper/2607.23124/) (2026)
- [SPIEval: Evaluating Large Language Models as Mobile Assistants over Scattered Personal Information](https://mlsanity.com/paper/2608.10692/) (2026)
- [CONTRAMEM: Learning Self-Evolving Procedural Memory from Contrasting Multi-Model Trajectories](https://mlsanity.com/paper/2608.22533/) (2026)
- [Root-Cause Attribution Is a Search Problem: Continual Search for Long-Horizon Agent Failures](https://mlsanity.com/paper/2609.13463/) (2026)
- [Trust and Task Completion in the World of Consumer AI Agents](https://mlsanity.com/paper/2609.33017/) (2026)
- [Self-Evolving Harness on Multiple Tasks with the Agent as Its Own Optimizer](https://mlsanity.com/paper/2609.38372/) (2026)
- [The Backdrop Exposes What the World Around an Agent Costs It](https://mlsanity.com/paper/2609.38469/) (2026)
- [ReLiveGym: Evaluating Long-Lived Agents over Weeks of Replayed Reality](https://mlsanity.com/paper/2610.00710/) (2026)
- [ActiveSaddler: Automated Curriculum Learning for Agent Harness Optimization](https://mlsanity.com/paper/2610.00906/) (2026)
- [ArrivalBench: Agent-Generated Data Pipelines Are Correct Once and Wrong Under Time](https://mlsanity.com/paper/2610.02363/) (2026)
- [LiveMACE: Process-Aware Evaluation of LLM Agent Capabilities in Evolving Markets](https://mlsanity.com/paper/2610.09872/) (2026)

## Abstract (arXiv)

We introduce Gaia2, a benchmark for evaluating large language model agents in realistic, asynchronous environments. Unlike prior static or synchronous evaluations, Gaia2 introduces scenarios where environments evolve independently of agent actions, requiring agents to operate under temporal constraints, adapt to noisy and dynamic events, resolve ambiguity, and collaborate with other agents. Each scenario is paired with a write-action verifier, enabling fine-grained, action-level evaluation and making Gaia2 directly usable for reinforcement learning from verifiable rewards. Our evaluation of state-of-the-art proprietary and open-source models shows that no model dominates across capabilities: GPT-5 (high) reaches the strongest overall score of 42% pass@1 but fails on time-sensitive tasks, Claude-4 Sonnet trades accuracy and speed for cost, Kimi-K2 leads among open-source models with 21% pass@1. These results highlight fundamental trade-offs between reasoning, efficiency, robustness, and expose challenges in closing the "sim2real" gap. Gaia2 is built on a consumer environment with the open-source Agents Research Environments platform and designed to be easy to extend. By releasing Gaia2 alongside the foundational ARE framework, we aim to provide the community with a flexible infrastructure for developing, benchmarking, and training the next generation of practical agent systems.