Reliable, Reproducible, and Really Fast Leaderboards with Evalica
Not on the map. Why?
Abstract (arXiv)
The rapid advancement of natural language processing (NLP) technologies, such as instruction-tuned large language models (LLMs), urges the development of modern evaluation protocols with human and machine feedback. We introduce Evalica, an open-source toolkit that facilitates the creation of reliable and reproducible model leaderboards. This paper presents its design, evaluates its performance, and demonstrates its usability through its Web interface, command-line interface, and Python API.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 6 of its 21 references that are in the map. First posted on arXiv 15 Dec 2024.
Builds on
6 of 21 references.
References · 6
- 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
- 2019Gradio: Hassle-Free Sharing and Testing of ML Models in the Wildcited by 176
- 2022Efficient computation of rankings from pairwise comparisonscited by 31
- 2021Learning from Crowds with Crowd-Kitcited by 23
- 2024Vikhr: The Family of Open-Source Instruction-Tuned Large Language Models for Russiancited by 12
- 2024OpenSkill: A faster asymmetric multi-team, multiplayer rating systemcited by 9
Built on by
Citing papers per year: 2025: 2; 2026: 1.
3 papers.