The Scaling Law of Evaluation Failure: Why Simple Averaging Collapses Under Data Sparsity and Item Difficulty Gaps, and How Item Response Theory Recovers Ground Truth Across Domains

Benchmark evaluation across AI and safety-critical domains overwhelmingly relies on simple averaging. We demonstrate that this practice produces substantially misleading rankings when two conditions co-occur: (1) the evaluation matrix is sparse and (2) items vary substantially in difficulty. Through controlled simulation experiments across four domains -- NLP (GLUE), clinical drug trials, autonomous vehicle safety, and cybersecurity -- we show that Spearman rank correlation ρ between simple-average rankings and ground-truth rankings degrades from ρ = 1.000 at 100% coverage to ρ = 0.809 at 67% coverage with high difficulty heterogeneity (mean over 20 seeds). A standard two-parameter logistic (2PL) Item Response Theory (IRT) model maintains ρ ≥ 0.996 across all conditions. A 150-condition grid sweep over sparsity S ∈ [0, 0.70] and difficulty gap D ∈ [0.5, 5.0] confirms that ranking error forms a failure surface with a strong S × D interaction (γ3 = +0.20, t = 13.05), while IRT maintains ρ ≥ 0.993 throughout. We discuss implications for Physical AI benchmarking, where evaluation matrices are often incomplete and difficulty gaps are extreme.