# Low Rank for Rank: Uncertainty-Aware Task-Specific LLM Ranking under Sparse Pairwise Comparisons

arXiv: https://arxiv.org/abs/2605.29395 · HTML: https://mlsanity.com/paper/2605.29395/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 35 references that are in the map.
- First posted on arXiv 28 May 2026.

## Builds on (most cited first, 6 of 6)

- [Dynamic Model Routing and Cascading for Efficient LLM Inference: A Survey](https://mlsanity.com/paper/2603.04445/) (2026)
- [Uncertainty Quantification for Large Language Model Reward Learning under Heterogeneous Human Feedback](https://mlsanity.com/paper/2512.03208/) (2025)
- [Uncertainty Quantification for Ranking with Heterogeneous Preferences](https://mlsanity.com/paper/2509.01847/) (2025)
- [Prompt-Dependent Ranking of Large Language Models with Uncertainty Quantification](https://mlsanity.com/paper/2603.03336/) (2026)
- [Evaluating LLMs When They Do Not Know the Answer: Statistical Evaluation of Mathematical Reasoning via Comparative Signals](https://mlsanity.com/paper/2602.03061/) (2026)
- [LLM Evaluation as Tensor Completion: Low Rank Structure and Semiparametric Efficiency](https://mlsanity.com/paper/2604.05460/) (2026)

## Built on by (most cited first, 2 of 2)

- [Rank Confidence Sequences:Anytime-valid Leaderboards](https://mlsanity.com/paper/2609.32211/) (2026)
- [Anytime-Valid LLM Leaderboards via Benchmark-weighted and Block-Factorized e-Processes](https://mlsanity.com/paper/2609.32248/) (2026)

## Abstract (arXiv)

Pairwise human-preference platforms such as Chatbot Arena have become central to large language model (LLM) evaluation, yet reliable task-specific ranking remains challenging. Global leaderboards mask task heterogeneity, while ranking each fine-grained task independently is unstable under sparse, imbalanced comparisons. We propose a low-rank framework for task-specific LLM ranking from sparse pairwise comparisons, modeling the task-by-model ability matrix Theta^star ∈ ℝdt × dm as low rank so that information is shared across related tasks while task-specific differences are preserved. We first develop a max-norm (ℓ_∞) accurate estimator for the latent scores, combining a convex initializer with alternating-minimization refinement, and prove task-wise top-K recovery guarantees under sparse sampling. Our main contribution is an uncertainty quantification framework for task-specific ranking. We construct cross-fitted one-step debiased estimators for fixed score contrasts -- such as the task-specific ability gap between two models -- yielding asymptotically valid confidence intervals that attain the semiparametric efficiency bound. We then extend the inference to the high-dimensional ranking regime, where per-task ranks and top-K membership are determined by many dependent score-gap hypotheses. Using Gaussian and multiplier-bootstrap calibration, we obtain simultaneous confidence sets for per-task ranks and valid top-K membership tests across many tasks and models. Experiments on synthetic data and Chatbot Arena show that low-rank sharing improves sample efficiency over independent task-wise Bradley-Terry estimation and produces tighter, better-calibrated ranking certificates, with the largest gains in the sparse regime typical of real LLM benchmarks.