# Reliable, Reproducible, and Really Fast Leaderboards with Evalica

arXiv: https://arxiv.org/abs/2412.11314 · HTML: https://mlsanity.com/paper/2412.11314/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 21 references that are in the map.
- First posted on arXiv 15 Dec 2024.

## Builds on (most cited first, 6 of 6)

- [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://mlsanity.com/paper/2403.04132/) (2024)
- [Gradio: Hassle-Free Sharing and Testing of ML Models in the Wild](https://mlsanity.com/paper/1906.02569/) (2019)
- [Efficient computation of rankings from pairwise comparisons](https://mlsanity.com/paper/2207.00076/) (2022)
- [Learning from Crowds with Crowd-Kit](https://mlsanity.com/paper/2109.08584/) (2021)
- [Vikhr: The Family of Open-Source Instruction-Tuned Large Language Models for Russian](https://mlsanity.com/paper/2405.13929/) (2024)
- [OpenSkill: A faster asymmetric multi-team, multiplayer rating system](https://mlsanity.com/paper/2401.05451/) (2024)

## Built on by (most cited first, 3 of 3)

- [Confidence and Stability of Global and Pairwise Scores in NLP Evaluation](https://mlsanity.com/paper/2507.01633/) (2025)
- [JointRank: Rank Large Set with Single Pass](https://mlsanity.com/paper/2506.22262/) (2025)
- [lmfaoooo at SemEval-2026 Task 1: Humor Is an Audience. Preference Modeling for Constrained Humor Generation](https://mlsanity.com/paper/2606.00022/) (2026)

## Abstract (arXiv)

The rapid advancement of natural language processing (NLP) technologies, such as instruction-tuned large language models (LLMs), urges the development of modern evaluation protocols with human and machine feedback. We introduce Evalica, an open-source toolkit that facilitates the creation of reliable and reproducible model leaderboards. This paper presents its design, evaluates its performance, and demonstrates its usability through its Web interface, command-line interface, and Python API.