mlsanityData to 14 Oct 2026

GAIA: a benchmark for General AI Assistants

arXiv 2311.12983 ↗First posted 21 Nov 2023cs.CL cs.AIcited by 1,128

Not on the map. Why?

Cited by 773 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce GAIA, a benchmark for General AI Assistants that, if solved, would represent a milestone in AI research. GAIA proposes real-world questions that require a set of fundamental abilities such as reasoning, multi-modality handling, web browsing, and generally tool-use proficiency. GAIA questions are conceptually simple for humans yet challenging for most advanced AIs: we show that human respondents obtain 92% vs. 15% for GPT-4 equipped with plugins. This notable performance disparity contrasts with the recent trend of LLMs outperforming humans on tasks requiring professional skills in e.g. law or chemistry. GAIA's philosophy departs from the current trend in AI benchmarks suggesting to target tasks that are ever more difficult for humans. We posit that the advent of Artificial General Intelligence (AGI) hinges on a system's capability to exhibit similar robustness as the average human does on such questions. Using GAIA's methodology, we devise 466 questions and their answer. We release our questions while retaining answers to 300 of them to power a leader-board available at https://huggingface.co/gaia-benchmark.

Key facts. Cited by 1,128 papers in the map (data through 14 Oct 2026). Builds on 23 of its 40 references that are in the map. First posted on arXiv 21 Nov 2023.

Builds on

23 of 40 references.

References · 23

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  6. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  8. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  9. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  10. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,332
  11. 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
  12. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 668
11 more
  1. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 603
  2. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  3. 2023TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answeringcited by 428
  4. 2023Multi-Agent Collaboration: Harnessing the Power of Intelligent LLM Agentscited by 333
  5. 2023ToolQA: A Dataset for LLM Question Answering with External Toolscited by 274
  6. 2023Auto-GPT for Online Decision Making: Benchmarks and Additional Opinionscited by 255
  7. 2022BlenderBot 3: a deployed conversational agent that continually learns to responsibly engagecited by 198
  8. 2023On the Tool Manipulation Capability of Open-source Large Language Modelscited by 122
  9. 2023AssistGPT: A General Multi-modal Assistant that can Plan, Execute, Inspect, and Learncited by 97
  10. 2023BOLAA: Benchmarking and Orchestrating LLM-augmented Autonomous Agentscited by 84
  11. 2023Gentopia: A Collaborative Platform for Tool-Augmented LLMscited by 26

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 94; 2025: 298; 2026: 733.

1,128 papers.

Sort:

By area:

  1. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  2. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,195
  3. 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agentscited by 1,002
  4. 2025BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agentscited by 647
  5. 2025Why Do Multi-Agent LLM Systems Fail?cited by 604
  6. 2024MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineeringcited by 469
  7. 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 468
  8. 2024WebVoyager: Building an End-to-End Web Agent with Large Multimodal Modelscited by 438
  9. 2024AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agentscited by 437
  10. 2024AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agentscited by 339
  11. 2025WebThinker: Empowering Large Reasoning Models with Deep Research Capabilitycited by 335
  12. 2025Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactionscited by 254
38 more
  1. 2025DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agentscited by 248
  2. 2024Magentic-One: A Generalist Multi-Agent System for Solving Complex Taskscited by 232
  3. 2025WebSailor: Navigating Super-human Reasoning for Web Agentcited by 229
  4. 2024AI Agents That Mattercited by 219
  5. 2024WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wildcited by 218
  6. 2024OS-Copilot: Towards Generalist Computer Agents with Self-Improvementcited by 205
  7. 2025Survey on Evaluation of LLM-based Agentscited by 204
  8. 2024Trial and Error: Exploration-Based Trajectory Optimization for LLM Agentscited by 197
  9. 2025WebDancer: Towards Autonomous Information Seeking Agencycited by 188
  10. 2025Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systemscited by 185
  11. 2025Multi-agent Architecture Search via Agentic Supernetcited by 179
  12. 2025MultiAgentBench: Evaluating the Collaboration and Competition of LLM agentscited by 178
  13. 2024RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human expertscited by 175
  14. 2025WebWalker: Benchmarking LLMs in Web Traversalcited by 175
  15. 2025Agentic Reinforced Policy Optimizationcited by 172
  16. 2024Teaching Large Language Models to Reason with Reinforcement Learningcited by 160
  17. 2024AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks?cited by 158
  18. 2025Measuring AI Ability to Complete Long Software Taskscited by 156
  19. 2026Advancing Open-source World Modelscited by 154
  20. 2024RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learningcited by 151
  21. 2025Beyond Ten Turns: Unlocking Long-Horizon Agentic Search with Large-Scale Asynchronous RLcited by 150
  22. 2025From LLM Reasoning to Autonomous AI Agents: A Comprehensive Reviewcited by 149
  23. 2025BrowseComp-ZH: Benchmarking Web Browsing Ability of Large Language Models in Chinesecited by 145
  24. 2025ReSum: Unlocking Long-Horizon Search Intelligence via Context Summarizationcited by 141
  25. 2025Where LLM Agents Fail and How They can Learn From Failurescited by 133
  26. 2024Exploring Large Language Model based Intelligent Agents: Definitions, Methods, and Prospectscited by 132
  27. 2025OWL: Optimized Workforce Learning for General Multi-Agent Assistance in Real-World Task Automationcited by 131
  28. 2024Observational Scaling Laws and the Predictability of Language Model Performancecited by 130
  29. 2026SoK: Agentic Skills -- Beyond Tool Use in LLM Agentscited by 130
  30. 2025Deep Research Agents: A Systematic Examination And Roadmapcited by 128
  31. 2025Memento: Fine-tuning LLM Agents without Fine-tuning LLMscited by 127
  32. 2025A Survey of Context Engineering for Large Language Modelscited by 126
  33. 2025MLGym: A New Framework and Benchmark for Advancing AI Research Agentscited by 124
  34. 2024WebCanvas: Benchmarking Web Agents in Online Environmentscited by 121
  35. 2025WebShaper: Agentically Data Synthesizing via Information-Seeking Formalizationcited by 121
  36. 2025WebWatcher: Breaking New Frontier of Vision-Language Deep Research Agentcited by 119
  37. 2024Empowering Biomedical Discovery with AI Agentscited by 118
  38. 2024Cradle: Empowering Foundation Agents Towards General Computer Controlcited by 117

50 most cited of 1,128 All 1,128 (JSON)