mlsanityData to 7 Oct 2026

Telco-GAIA: Bilingual Benchmark for Agents in Telecom Domain

arXiv 2607.20510 ↗First posted 4 Jul 2026cs.AI cs.CLcited by 0

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

We introduce Telco-GAIA, a bilingual, multi-modal benchmark for evaluating tool-using agents on the data of a real-world telecommunications operator. Telco-GAIA comprises 100 human-verified question-answering tasks, in English and Arabic, that each demand multi-hop reasoning (4.2 hops on average) over three heterogeneous sources: a static website snapshot (HTML, images, and linked PDFs), a synthetic relational SQL database, and external web archives, spanning text, image, and tabular modalities. The benchmark is delivered as a sandboxed Docker environment and scored by normalized exact string matching, making evaluation objective, deterministic, and reproducible over time without any LLM-as-a-Judge. Evaluating a purpose-built reference agent across twelve commercial and open LLMs, we find Telco-GAIA challenging: even the strongest model solves only 71% of tasks; under a moderate cost budget, this falls to about 40%, and the visually grounded categories remain the weakest, where the average backend scores below 30%, leaving substantial headroom in document and image understanding. Telco-GAIA offers a rigorous, reproducible testbed for enterprise agents and a template for constructing closed-domain benchmarks.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 27 of its 43 references that are in the map. First posted on arXiv 4 Jul 2026.

Builds on

27 of 43 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks18 Dec
  2. 2024How to Correctly do Semantic Backpropagation on Language-based Agentic Systems4 Dec
  3. 2024Agent-as-a-Judge: Evaluate Agents with Agents14 Oct
  4. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains17 Jun

Other references · 23

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  3. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  4. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
  5. 2023GAIA: a benchmark for General AI Assistantscited by 1,084
  6. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  7. 2025BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agentscited by 616
  8. 2020AmbigQA: Answering Ambiguous Open-domain Questionscited by 385
  9. 2024WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?cited by 348
  10. 2023FinanceBench: A New Benchmark for Financial Question Answeringcited by 261
  11. 2023ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systemscited by 259
  12. 2024MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queriescited by 253
11 more
  1. 2024MMLongBench-Doc: Benchmarking Long-context Document Understanding with Visualizationscited by 174
  2. 2024Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generationcited by 158
  3. 2024AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks?cited by 155
  4. 2024BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrievalcited by 153
  5. 2024RAGBench: Explainable Benchmark for Retrieval-Augmented Generation Systemscited by 115
  6. 2021NeurIPS 2020 EfficientQA Competition: Systems, Analyses and Lessons Learnedcited by 52
  7. 2025ToolHop: A Query-Driven Benchmark for Evaluating Large Language Models in Multi-Hop Tool Usecited by 48
  8. 2026τ-Knowledge: Evaluating Conversational Agents over Unstructured Knowledgecited by 35
  9. 2024MINTQA: A Multi-Hop Question Answering Benchmark for Evaluating LLMs on New and Tail Knowledgecited by 20
  10. 2024SEC-QA: A Systematic Evaluation Corpus for Financial QAcited by 17
  11. 2025LiveRAG: A diverse Q&A dataset with varying difficulty level for RAG evaluationcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026TelcoAgent-Bench: A Multilingual Benchmark for Telecom AI Agents
  2. 2026H2LooP Telecom Model v1: From Telecom Comprehension to Autonomous Issue and PR Resolution
  3. 2025Evaluating Open-Source Large Language Models for Technical Telecom Question Answering
  4. 2025RAGCap-Bench: Benchmarking Capabilities of LLMs in Agentic Retrieval Augmented Generation Systems
  5. 2025TeleMath: A Benchmark for Large Language Models in Telecom Mathematical Problem Solving
  6. 2026Two Calls Beat Five Agents: Evaluating Multi-Agent Pipelines Against Self-Refinement for Local Language Models
  7. 2026A Tri-Agent Framework for Evaluating and Aligning Question Clarification Capabilities of Large Language Models
  8. 2025Ticket-Bench: A Kickoff for Multilingual and Regionalized Agent Evaluation