mlsanityData to 7 Oct 2026

EvalAI: Towards Better Evaluation Systems for AI Agents

arXiv 1902.03570 ↗First posted 10 Feb 2019cs.AI cs.CL cs.CVcited by 34

Not on the map. Why?

Abstract (arXiv)

We introduce EvalAI, an open source platform for evaluating and comparing machine learning (ML) and artificial intelligence algorithms (AI) at scale. EvalAI is built to provide a scalable solution to the research community to fulfill the critical need of evaluating machine learning models and agents acting in an environment against annotations or with a human-in-the-loop. This will help researchers, students, and data scientists to create, collaborate, and participate in AI challenges organized around the globe. By simplifying and standardizing the process of benchmarking these models, EvalAI seeks to lower the barrier to entry for participating in the global scientific effort to push the frontiers of machine learning and artificial intelligence, thereby increasing the rate of measurable progress in this domain.

Key facts. Cited by 34 papers in the map (data through 7 Oct 2026). First posted on arXiv 10 Feb 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 6; 2022: 5; 2023: 8; 2024: 5; 2025: 7; 2026: 3.

34 papers.

Sort:

By area:

  1. 2022ProcTHOR: Large-Scale Embodied AI Using Procedural Generationcited by 369
  2. 2021Hurdles to Progress in Long-form Question Answeringcited by 166
  3. 2023AM-RADIO: Agglomerative Vision Foundation Model -- Reduce All Domains Into Onecited by 165
  4. 2024BRAVE: Broadening the visual encoding of vision-language modelscited by 68
  5. 2021Neural Latents Benchmark '21: Evaluating latent variable models of neural population activitycited by 57
  6. 2023Interactive Natural Language Processingcited by 31
  7. 2022Evaluate & Evaluation on the Hub: Better Best Practices for Data and Model Measurementscited by 21
  8. 2023Alexa Arena: A User-Centric Interactive Platform for Embodied AIcited by 18
  9. 2023Resources for Brewing BEIR: Reproducible Reference Models and an Official Leaderboardcited by 15
  10. 2025A Survey on Agent Workflow -- Status and Futurecited by 14
  11. 2021VISITRON: Visual Semantics-Aligned Interactively Trained Object-Navigatorcited by 13
  12. 2022MIA 2022 Shared Task: Evaluating Cross-lingual Open-Retrieval Question Answering for 16 Diverse Languagescited by 11
22 more
  1. 2024The VoxCeleb Speaker Recognition Challenge: A Retrospectivecited by 9
  2. 2025The Interspeech 2025 Speech Accessibility Project Challengecited by 9
  3. 2025Second Competition on Presentation Attack Detection on ID Cardcited by 8
  4. 2021NeurIPS 2020 NLC2CMD Competition: Translating Natural Language to Bash Commandscited by 7
  5. 2023ICDAR 2023 Competition on Robust Layout Segmentation in Corporate Documentscited by 6
  6. 2023DR-Label: Improving GNN Models for Catalysis Systems by Label Deconstruction and Reconstructioncited by 4
  7. 2025Do Large Multimodal Models Solve Caption Generation for Scientific Figures? Lessons Learned from SciCap Challenge 2023cited by 4
  8. 2022The Massively Multilingual Natural Language Understanding 2022 (MMNLU-22) Workshop and Competitioncited by 3
  9. 2024EXACT: Towards a platform for empirically benchmarking Machine Learning model explanation methodscited by 3
  10. 2026The Behavioral Fabric of LLM-Powered GUI Agents: Human Values and Interaction Outcomescited by 3
  11. 2021ECOL-R: Encouraging Copying in Novel Object Captioning with Reinforcement Learningcited by 2
  12. 2023DiagGPT: An LLM-based and Multi-agent Dialogue System with Automatic Topic Management for Flexible Task-Oriented Dialoguecited by 2
  13. 2025Towards General Urban Monitoring with Vision-Language Models: A Review, Evaluation, and a Research Agendacited by 2
  14. 2022Open-Source Framework for Encrypted Internet and Malicious Traffic Classificationcited by 1
  15. 2023Atrial Septal Defect Detection in Children Based on Ultrasound Video Using Multiple Instances Learningcited by 1
  16. 2024HomeRobot Open Vocabulary Mobile Manipulation Challenge 2023 Participant Report (Team KuzHum)cited by 1
  17. 2024PPLqa: An Unsupervised Information-Theoretic Quality Metric for Comparing Generative Large Language Modelscited by 1
  18. 2026Video Active Perception: Effective Inference-Time Long-Form Video Understanding with Vision-Language Modelscited by 1
  19. 2021A Novel Methodology For Crowdsourcing AI Models in an Enterprisecited by 0
  20. 2025Machine-learning competition to grade EEG background patterns in newborns with hypoxic-ischaemic encephalopathycited by 0
  21. 2025Challenge on Optimization of Context Collection for Code Completioncited by 0
  22. 2026Fusion4CA: Boosting 3D Object Detection via Comprehensive Image Exploitationcited by 0