# BabelArena: A Large-Scale Multilingual Benchmark for LLM Agents

arXiv: https://arxiv.org/abs/2609.23490 · HTML: https://mlsanity.com/paper/2609.23490/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use.
- Builds on 11 of its 22 references that are in the map.
- First posted on arXiv 20 Sep 2026.

## Placed in

- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)

## Builds on (most cited first, 11 of 11)

- [τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains](https://mlsanity.com/paper/2406.12045/) (2024)
- [AgentBench: Evaluating LLMs as Agents](https://mlsanity.com/paper/2308.03688/) (2023)
- [τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environment](https://mlsanity.com/paper/2506.07982/) (2025)
- [The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants](https://mlsanity.com/paper/2308.16884/) (2023)
- [VitaBench: Benchmarking LLM Agents with Versatile Interactive Tasks in Real-world Applications](https://mlsanity.com/paper/2509.26490/) (2025)
- [DeepPlanning: Benchmarking Long-Horizon Agentic Planning with Verifiable Constraints](https://mlsanity.com/paper/2601.18137/) (2026)
- [τ-Knowledge: Evaluating Conversational Agents over Unstructured Knowledge](https://mlsanity.com/paper/2603.04370/) (2026)
- [Lost in Execution: On the Multilingual Robustness of Tool Calling in Large Language Models](https://mlsanity.com/paper/2601.05366/) (2026)
- [KV-PRM: Efficient Process Reward Modeling via KV-Cache Transfer for Multi-Agent Test-Time Scaling](https://mlsanity.com/paper/2607.09153/) (2026)
- [GAIA-v2-LILT: Multilingual Adaptation of Agent Benchmark beyond Translation](https://mlsanity.com/paper/2604.24929/) (2026)
- [OmnilingualGAIA2: Evaluating the Multilingual Gap in Frontier AI Agents](https://mlsanity.com/paper/2608.08775/) (2026)

## Built on by (most cited first, 1 of 1)

- [When Agent Evaluation Becomes Unreliable: An Empirical Study of Oracle Failures and Configuration-Dependent Measurement in an LLM Agent Benchmark](https://mlsanity.com/paper/2610.16067/) (2026)

## Abstract (arXiv)

Large language model (LLM) agents increasingly execute multi-step workflows through tool use and interaction with users and environments. However, current agent evaluations are largely English-centric, limiting our understanding of agent capabilities in multilingual settings. We introduce BabelFlow, a benchmark-general agentic workflow that adapts existing agent benchmarks to new languages by analyzing runtime dependencies, coordinating structure-preserving translation, and combining multi-layer verification with human review to preserve task and evaluation semantics. Using BabelFlow, we construct BabelArena, a task-aligned benchmark comprising 16,146 instances derived from 702 canonical tasks across four benchmark families, 13 domains, and 23 languages. Experiments with five frontier models show that no single model dominates across benchmark families and that cross-language disparities extend well beyond task success. Lower-resource languages exhibit distinct failure patterns, with larger shares of tool-use and control-flow errors rather than answer-quality errors alone, pointing to gaps in reliable task execution across the resource levels of these languages. On the same tasks, agents in low-resource languages also consume substantially more tokens than in English (up to roughly twice the input) without proportional increases in interaction length, and language consistency degrades further on tasks requiring structured output, where switches are directed overwhelmingly toward English. We believe BabelArena provides a foundation for advancing research on reliable and efficient multilingual agents.