Identity-Link IRT for Label-Free LLM Evaluation: Preserving Additivity in TVD-MI Scores
Not on the map. Why?
Abstract (arXiv)
Pairwise comparisons of large language models using total variation distance mutual information (TVD-MI) produce binary critic decisions per pair. We show that averaging TVD-MI's binary trials yields centered-probability scores with additive structure suitable for item-response theory (IRT) without nonlinear link functions. Maximum-likelihood approaches to IRT use logistic links, but we find empirically that these transformations introduce curvature that breaks additivity: across three domains, the identity link yields median curl on raw data of 0.080-0.150 (P95 = [0.474, 0.580]), whereas probit/logit introduce substantially higher violations (median [0.245, 0.588], P95 [0.825, 2.252]). We derive this clipped-linear model from Gini entropy maximization, yielding a box-constrained least-squares formulation that handles boundary saturation. At 33% coverage, we achieve holdout RMSE 0.117 ± 0.008 while preserving agent rankings (Spearman ρ = 0.972 ± 0.015), three times fewer evaluations than full dense. Judge robustness analysis (GPT-4o-mini vs. Llama3-70b) shows strong agreement in agent rankings (ρ = 0.872) and consistent identity-link advantage. TVD-MI's geometry is best preserved by identity mapping for efficient LLM evaluation, applicable to other bounded-response domains.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 6 of its 17 references that are in the map. First posted on arXiv 16 Oct 2025.
Builds on
6 of 17 references.
References · 6
- 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
- 2025Lost in Benchmarks? Rethinking Large Language Model Benchmarking with Item Response Theorycited by 45
- 2024Leveraging LLM-Respondents for Item Evaluation: a Psychometric Analysiscited by 31
- 2013Crowdsourced Judgement Elicitation with Endogenous Proficiencycited by 30
- 2024Easy2Hard-Bench: Standardized Difficulty Labels for Profiling LLM Performance and Generalizationcited by 24
- 2024Benchmarking LLMs' Judgments with No Gold Standardcited by 19
Built on by
Citing papers per year: 2026: 2.
2 papers.
- 2026Can We Trust Item Response Theory for AI Evaluation?16 Jul 2026
- 2026DualEval: Joint Model-Item Calibration for Unified LLM Evaluation24 Jun 2026