# EvalAI: Towards Better Evaluation Systems for AI Agents

arXiv: https://arxiv.org/abs/1902.03570 · HTML: https://mlsanity.com/paper/1902.03570/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Feb 2019.

## Built on by (most cited first, 30 of 34)

- [ProcTHOR: Large-Scale Embodied AI Using Procedural Generation](https://mlsanity.com/paper/2206.06994/) (2022)
- [AM-RADIO: Agglomerative Vision Foundation Model -- Reduce All Domains Into One](https://mlsanity.com/paper/2312.06709/) (2023)
- [Hurdles to Progress in Long-form Question Answering](https://mlsanity.com/paper/2103.06332/) (2021)
- [BRAVE: Broadening the visual encoding of vision-language models](https://mlsanity.com/paper/2404.07204/) (2024)
- [Neural Latents Benchmark '21: Evaluating latent variable models of neural population activity](https://mlsanity.com/paper/2109.04463/) (2021)
- [Interactive Natural Language Processing](https://mlsanity.com/paper/2305.13246/) (2023)
- [Evaluate & Evaluation on the Hub: Better Best Practices for Data and Model Measurements](https://mlsanity.com/paper/2210.01970/) (2022)
- [Alexa Arena: A User-Centric Interactive Platform for Embodied AI](https://mlsanity.com/paper/2303.01586/) (2023)
- [Resources for Brewing BEIR: Reproducible Reference Models and an Official Leaderboard](https://mlsanity.com/paper/2306.07471/) (2023)
- [A Survey on Agent Workflow -- Status and Future](https://mlsanity.com/paper/2508.01186/) (2025)
- [VISITRON: Visual Semantics-Aligned Interactively Trained Object-Navigator](https://mlsanity.com/paper/2105.11589/) (2021)
- [MIA 2022 Shared Task: Evaluating Cross-lingual Open-Retrieval Question Answering for 16 Diverse Languages](https://mlsanity.com/paper/2207.00758/) (2022)
- [The VoxCeleb Speaker Recognition Challenge: A Retrospective](https://mlsanity.com/paper/2408.14886/) (2024)
- [The Interspeech 2025 Speech Accessibility Project Challenge](https://mlsanity.com/paper/2507.22047/) (2025)
- [Second Competition on Presentation Attack Detection on ID Card](https://mlsanity.com/paper/2507.20404/) (2025)
- [NeurIPS 2020 NLC2CMD Competition: Translating Natural Language to Bash Commands](https://mlsanity.com/paper/2103.02523/) (2021)
- [ICDAR 2023 Competition on Robust Layout Segmentation in Corporate Documents](https://mlsanity.com/paper/2305.14962/) (2023)
- [DR-Label: Improving GNN Models for Catalysis Systems by Label Deconstruction and Reconstruction](https://mlsanity.com/paper/2303.02875/) (2023)
- [Do Large Multimodal Models Solve Caption Generation for Scientific Figures? Lessons Learned from SciCap Challenge 2023](https://mlsanity.com/paper/2501.19353/) (2025)
- [The Behavioral Fabric of LLM-Powered GUI Agents: Human Values and Interaction Outcomes](https://mlsanity.com/paper/2601.16356/) (2026)
- [The Massively Multilingual Natural Language Understanding 2022 (MMNLU-22) Workshop and Competition](https://mlsanity.com/paper/2212.06346/) (2022)
- [EXACT: Towards a platform for empirically benchmarking Machine Learning model explanation methods](https://mlsanity.com/paper/2405.12261/) (2024)
- [ECOL-R: Encouraging Copying in Novel Object Captioning with Reinforcement Learning](https://mlsanity.com/paper/2101.09865/) (2021)
- [DiagGPT: An LLM-based and Multi-agent Dialogue System with Automatic Topic Management for Flexible Task-Oriented Dialogue](https://mlsanity.com/paper/2308.08043/) (2023)
- [Towards General Urban Monitoring with Vision-Language Models: A Review, Evaluation, and a Research Agenda](https://mlsanity.com/paper/2510.12400/) (2025)
- [Open-Source Framework for Encrypted Internet and Malicious Traffic Classification](https://mlsanity.com/paper/2206.10144/) (2022)
- [Atrial Septal Defect Detection in Children Based on Ultrasound Video Using Multiple Instances Learning](https://mlsanity.com/paper/2306.03835/) (2023)
- [HomeRobot Open Vocabulary Mobile Manipulation Challenge 2023 Participant Report (Team KuzHum)](https://mlsanity.com/paper/2401.12048/) (2024)
- [PPLqa: An Unsupervised Information-Theoretic Quality Metric for Comparing Generative Large Language Models](https://mlsanity.com/paper/2411.15320/) (2024)
- [Video Active Perception: Effective Inference-Time Long-Form Video Understanding with Vision-Language Models](https://mlsanity.com/paper/2605.01662/) (2026)

## Abstract (arXiv)

We introduce EvalAI, an open source platform for evaluating and comparing machine learning (ML) and artificial intelligence algorithms (AI) at scale. EvalAI is built to provide a scalable solution to the research community to fulfill the critical need of evaluating machine learning models and agents acting in an environment against annotations or with a human-in-the-loop. This will help researchers, students, and data scientists to create, collaborate, and participate in AI challenges organized around the globe. By simplifying and standardizing the process of benchmarking these models, EvalAI seeks to lower the barrier to entry for participating in the global scientific effort to push the frontiers of machine learning and artificial intelligence, thereby increasing the rate of measurable progress in this domain.