# Demystifying LLM-as-a-Judge: Analytically Tractable Model for Inference-Time Scaling

arXiv: https://arxiv.org/abs/2512.19905 · HTML: https://mlsanity.com/paper/2512.19905/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning.
- Builds on 24 of its 66 references that are in the map.
- First posted on arXiv 22 Dec 2025.

## Placed in

- LLM reasoning / [Test-time compute and efficient thinking](https://mlsanity.com/lr/test-time-compute-and-efficient-thinking/)

## Builds on (most cited first, 24 of 24)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [Deep Learning Scaling is Predictable, Empirically](https://mlsanity.com/paper/1712.00409/) (2017)
- [Generative Verifiers: Reward Modeling as Next-Token Prediction](https://mlsanity.com/paper/2408.15240/) (2024)
- [Self-critiquing models for assisting human evaluators](https://mlsanity.com/paper/2206.05802/) (2022)
- [Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Models](https://mlsanity.com/paper/2408.00724/) (2024)
- [Training Language Models to Reason Efficiently](https://mlsanity.com/paper/2502.04463/) (2025)
- [A Solvable Model of Neural Scaling Laws](https://mlsanity.com/paper/2210.16859/) (2022)
- [Is Best-of-N the Best of Them? Coverage, Scaling, and Optimality in Inference-Time Alignment](https://mlsanity.com/paper/2503.21878/) (2025)
- [Scaling and renormalization in high-dimensional regression](https://mlsanity.com/paper/2405.00592/) (2024)
- [How Do Large Language Monkeys Get Their Power (Laws)?](https://mlsanity.com/paper/2502.17578/) (2025)
- [More Data Can Hurt for Linear Regression: Sample-wise Double Descent](https://mlsanity.com/paper/1912.07242/) (2019)
- [A Simple Model of Inference Scaling Laws](https://mlsanity.com/paper/2410.16377/) (2024)
- [Optimizing Temperature for Language Models with Multi-Sample Inference](https://mlsanity.com/paper/2502.05234/) (2025)
- [Rethinking Fine-Tuning when Scaling Test-Time Compute: Limiting Confidence Improves Mathematical Reasoning](https://mlsanity.com/paper/2502.07154/) (2025)
- [A non-asymptotic theory of Kernel Ridge Regression: deterministic equivalents, test error, and GCV estimator](https://mlsanity.com/paper/2403.08938/) (2024)
- [The Eigenlearning Framework: A Conservation Law Perspective on Kernel Regression and Wide Neural Networks](https://mlsanity.com/paper/2110.03922/) (2021)
- [Sample, Don't Search: Rethinking Test-Time Alignment for Language Models](https://mlsanity.com/paper/2504.03790/) (2025)
- [Asymptotic learning curves of kernel methods: empirical data v.s. Teacher-Student paradigm](https://mlsanity.com/paper/1905.10843/) (2019)
- [Best-of-N through the Smoothing Lens: KL Divergence and Regret Analysis](https://mlsanity.com/paper/2507.05913/) (2025)
- [Sample Complexity and Representation Ability of Test-time Scaling Paradigms](https://mlsanity.com/paper/2506.05295/) (2025)
- [Best-of-Majority: Minimax-Optimal Strategy for Pass@k Inference Scaling](https://mlsanity.com/paper/2510.03199/) (2025)

## Built on by (most cited first, 4 of 4)

- [Jailbreak Scaling Laws for Large Language Models: Polynomial-Exponential Crossover](https://mlsanity.com/paper/2603.11331/) (2026)
- [Predicting and improving test-time scaling laws via reward tail-guided search](https://mlsanity.com/paper/2602.01485/) (2026)
- [Partially Correlated Verifier Cascades in LLM Harnesses: Concave Log-Odds, Polynomial Reliability, and Blind-Spot Ceilings](https://mlsanity.com/paper/2607.13918/) (2026)
- [An Asymptotic Theory of Chain-of-Thought in In-Context Learning](https://mlsanity.com/paper/2606.03217/) (2026)

## Abstract (arXiv)

Recent developments in large language models have shown advantages in reallocating a notable share of computational resource from training time to inference time. However, the principles behind inference time scaling are not well understood. In this paper, we introduce an analytically tractable model of inference-time scaling: Bayesian linear regression with a reward-weighted sampler, where the reward is determined from a linear model, modeling LLM-as-a-judge scenario. We study this problem in the high-dimensional regime, where the deterministic equivalents dictate a closed-form expression for the posterior predictive mean and variance. We analyze the generalization error when training data are sampled from a teacher model. We draw k inference-time samples and select via softmax at a temperature applied to a quadratic reward. When the reward is not too different from the teacher, the generalization error decreases monotonically with increasing inference time samples k. However, the specific reward that optimizes inference-time selection generally differs from the teacher. In contrast, substantial reward misspecification induces a finite optimal k beyond which more sampling can increase the generalization error. For fixed k, there exists an optimal sampling temperature. We experimentally verify these facts in large language model inference with an additional large language model as a judge. In the "best-of-k" limit with the teacher as reward, we theoretically show that the generalization error decays as Theta(1/k2) and determine the leading coefficient via extreme value theory. These formulas delineate domains where scaling inference-time computation is provably preferable to collecting more data. Finally, we demonstrate that when task difficulty increases, the previously mentioned advantage of inference-time compute degrades.