# GAIA-v2-LILT: Multilingual Adaptation of Agent Benchmark beyond Translation

arXiv: https://arxiv.org/abs/2604.24929 · HTML: https://mlsanity.com/paper/2604.24929/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 44 references that are in the map.
- First posted on arXiv 27 Apr 2026.

## Builds on (most cited first, 26 of 26)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023)
- [The Rise and Potential of Large Language Model Based Agents: A Survey](https://mlsanity.com/paper/2309.07864/) (2023)
- [WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents](https://mlsanity.com/paper/2207.01206/) (2022)
- [Mind2Web: Towards a Generalist Agent for the Web](https://mlsanity.com/paper/2306.06070/) (2023)
- [OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments](https://mlsanity.com/paper/2404.07972/) (2024)
- [GAIA: a benchmark for General AI Assistants](https://mlsanity.com/paper/2311.12983/) (2023)
- [BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents](https://mlsanity.com/paper/2504.12516/) (2025)
- [τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environment](https://mlsanity.com/paper/2506.07982/) (2025)
- [MEGA: Multilingual Evaluation of Generative AI](https://mlsanity.com/paper/2303.12528/) (2023)
- [CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languages](https://mlsanity.com/paper/2309.09400/) (2023)
- [GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Tasks](https://mlsanity.com/paper/2510.04374/) (2025)
- [BrowseComp-ZH: Benchmarking Web Browsing Ability of Large Language Models in Chinese](https://mlsanity.com/paper/2504.19314/) (2025)
- [Branch-Solve-Merge Improves Large Language Model Evaluation and Generation](https://mlsanity.com/paper/2310.15123/) (2023)
- [Don't Trust ChatGPT when Your Question is not in English: A Study of Multilingual Abilities and Types of LLMs](https://mlsanity.com/paper/2305.16339/) (2023)
- [CheckEval: A reliable LLM-as-a-Judge framework for evaluating text generation using checklists](https://mlsanity.com/paper/2403.18771/) (2024)
- [Lost in Multilinguality: Dissecting Cross-lingual Factual Inconsistency in Transformer Language Models](https://mlsanity.com/paper/2504.04264/) (2025)
- [mHumanEval -- A Multilingual Benchmark to Evaluate Large Language Models for Code Generation](https://mlsanity.com/paper/2410.15037/) (2024)
- [Contextual Refinement of Translations: Large Language Models for Sentence and Document-Level Post-Editing](https://mlsanity.com/paper/2310.14855/) (2023)
- [M-MAD: Multidimensional Multi-Agent Debate for Advanced Machine Translation Evaluation](https://mlsanity.com/paper/2412.20127/) (2024)
- [Lost in Literalism: How Supervised Training Shapes Translationese in LLMs](https://mlsanity.com/paper/2503.04369/) (2025)
- [Translation as a Scalable Proxy for Multilingual Evaluation](https://mlsanity.com/paper/2601.11778/) (2026)
- [Ticket-Bench: A Kickoff for Multilingual and Regionalized Agent Evaluation](https://mlsanity.com/paper/2509.14477/) (2025)
- [SEADialogues: A Multilingual Culturally Grounded Multi-turn Dialogue Dataset on Southeast Asian Languages](https://mlsanity.com/paper/2508.07069/) (2025)

## Built on by (most cited first, 2 of 2)

- [An Actionable Diagnosis of Multilingual, Multi-Agent Planning Failures](https://mlsanity.com/paper/2608.03735/) (2026)
- [BabelArena: A Large-Scale Multilingual Benchmark for LLM Agents](https://mlsanity.com/paper/2609.23490/) (2026)

## Abstract (arXiv)

Agent benchmarks remain largely English-centric, while their multilingual versions are often built with machine translation (MT) and limited post-editing. We argue that, for agentic tasks, this minimal workflow can easily break benchmark validity through query-answer misalignment or culturally off-target context. We propose a refined workflow for adapting English benchmarks into multiple languages with explicit functional alignment, cultural alignment, and difficulty calibration using both automated checks and human review. Using this workflow, we introduce GAIA-v2-LILT, a re-audited multilingual extension of GAIA covering five non-English languages. In experiments, our workflow improves agent success rates by up to 32.7% over minimally translated versions, bringing the closest audited setting to within 3.1% of English performance while substantial gaps remain in many other cases. This indicates that a substantial share of the multilingual performance gap is benchmark-induced measurement error, motivating task-level alignment when adapting English benchmarks across languages. The data is available as part of the MAPS package at https://huggingface.co/datasets/Fujitsu-FRE/MAPS/viewer/GAIA-v2-LILT. We also release the code used in our experiments at https://github.com/lilt/gaia-v2-lilt.