mlsanityData to 14 Oct 2026

Identity-Link IRT for Label-Free LLM Evaluation: Preserving Additivity in TVD-MI Scores

arXiv 2510.14966 ↗First posted 16 Oct 2025cs.LG stat.MLcited by 2

Not on the map. Why?

Abstract (arXiv)

Pairwise comparisons of large language models using total variation distance mutual information (TVD-MI) produce binary critic decisions per pair. We show that averaging TVD-MI's binary trials yields centered-probability scores with additive structure suitable for item-response theory (IRT) without nonlinear link functions. Maximum-likelihood approaches to IRT use logistic links, but we find empirically that these transformations introduce curvature that breaks additivity: across three domains, the identity link yields median curl on raw data of 0.080-0.150 (P95 = [0.474, 0.580]), whereas probit/logit introduce substantially higher violations (median [0.245, 0.588], P95 [0.825, 2.252]). We derive this clipped-linear model from Gini entropy maximization, yielding a box-constrained least-squares formulation that handles boundary saturation. At 33% coverage, we achieve holdout RMSE 0.117 ± 0.008 while preserving agent rankings (Spearman ρ = 0.972 ± 0.015), three times fewer evaluations than full dense. Judge robustness analysis (GPT-4o-mini vs. Llama3-70b) shows strong agreement in agent rankings (ρ = 0.872) and consistent identity-link advantage. TVD-MI's geometry is best preserved by identity mapping for efficient LLM evaluation, applicable to other bounded-response domains.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 6 of its 17 references that are in the map. First posted on arXiv 16 Oct 2025.

Builds on

6 of 17 references.

References · 6

  1. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  2. 2025Lost in Benchmarks? Rethinking Large Language Model Benchmarking with Item Response Theorycited by 45
  3. 2024Leveraging LLM-Respondents for Item Evaluation: a Psychometric Analysiscited by 31
  4. 2013Crowdsourced Judgement Elicitation with Endogenous Proficiencycited by 30
  5. 2024Easy2Hard-Bench: Standardized Difficulty Labels for Profiling LLM Performance and Generalizationcited by 24
  6. 2024Benchmarking LLMs' Judgments with No Gold Standardcited by 19

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.