# Scorio.jl: A Julia package for ranking stochastic responses

arXiv: https://arxiv.org/abs/2603.14103 · HTML: https://mlsanity.com/paper/2603.14103/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 7 of its 42 references that are in the map.
- First posted on arXiv 14 Mar 2026.

## Builds on (most cited first, 7 of 7)

- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [α-Rank: Multi-Agent Evaluation by Evolution](https://mlsanity.com/paper/1903.01373/) (2019)
- [Re-evaluating Evaluation](https://mlsanity.com/paper/1806.02643/) (2018)
- [Don't Pass@k: A Bayesian Framework for Large Language Model Evaluation](https://mlsanity.com/paper/2510.04265/) (2025)
- [Spectral Ranking using Seriation](https://mlsanity.com/paper/1406.5370/) (2014)
- [Davidson-Luce model for multi-item choice with ties](https://mlsanity.com/paper/1909.07123/) (2019)
- [Ranking Reasoning LLMs under Test-Time Scaling](https://mlsanity.com/paper/2603.10960/) (2026)

## Abstract (arXiv)

Scorio.jl is a Julia package for evaluating and ranking systems from repeated responses to shared tasks. It provides a common tensor-based interface for direct score-based, pairwise, psychometric, voting, graph, and listwise methods, so the same benchmark can be analyzed under multiple ranking assumptions. We describe the package design, position it relative to existing Julia tools, and report pilot experiments on synthetic rank recovery, stability under limited trials, and runtime scaling.