mlsanityData to 14 Oct 2026

Quantitative LLM Judges

arXiv 2506.02945 ↗First posted 3 Jun 2025cs.CL cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

LLM-as-a-judge is a framework where a large language model (LLM) evaluates the output of another LLM. While LLMs excel at producing qualitative textual evaluations, they often struggle to predict human preferences and numeric scores. We propose quantitative LLM judges, which align evaluation scores of existing LLM judges to humans in a given domain using regression models. The models are trained to improve the score of the original judge using its rationale and score. We present four quantitative judges for different types of absolute and relative feedback, which showcases the generality and versatility of our framework. Our framework is more computationally efficient than supervised fine-tuning and can be more statistically efficient when human feedback is limited, which is expected in practice. We validate these claims empirically on four datasets using two base judges. Our experiments show that quantitative judges can improve the predictive power of existing judges through post-hoc modeling.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 41 of its 54 references that are in the map. First posted on arXiv 3 Jun 2025.

Builds on

41 of 54 references.

References · 41

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  3. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  4. 2020Learning to summarize from human feedbackcited by 2,588
  5. 2024A Survey on LLM-as-a-Judgecited by 1,423
  6. 2016Understanding intermediate layers using linear classifier probescited by 1,353
  7. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  8. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 974
  9. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 876
  10. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 812
  11. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 734
  12. 2024Self-Rewarding Language Modelscited by 681
29 more
  1. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 461
  2. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 446
  3. 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 367
  4. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 318
  5. 2023PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimizationcited by 316
  6. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 302
  7. 2024LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Taskscited by 243
  8. 2023ToxicChat: Unveiling Hidden Challenges of Toxicity Detection in Real-World User-AI Conversationcited by 231
  9. 2024SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Modelscited by 228
  10. 2024Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judgescited by 205
  11. 2022Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Textcited by 155
  12. 2024HelpSteer2-Preference: Complementing Ratings with Preferencescited by 130
  13. 2025Preference Leakage: A Contamination Problem in LLM-as-a-judgecited by 125
  14. 2024Taming Overconfidence in LLMs: Reward Calibration in RLHFcited by 112
  15. 2024An Empirical Study of LLM-as-a-Judge for LLM Evaluation: Fine-tuned Judge Model is not a General Substitute for GPT-4cited by 94
  16. 2024OffsetBias: Leveraging Debiased Data for Tuning Evaluatorscited by 94
  17. 2024The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Modelscited by 93
  18. 2024Systematic Evaluation of LLM-as-a-Judge in LLM Alignment Tasks: Explainable Metrics and Diverse Prompt Templatescited by 72
  19. 2023CritiqueLLM: Towards an Informative Critique Generation Model for Evaluation of Large Language Model Generationcited by 66
  20. 2025Great Models Think Alike and this Undermines AI Oversightcited by 58
  21. 2024FLEUR: An Explainable Reference-Free Evaluation Metric for Image Captioning Using a Large Multimodal Modelcited by 45
  22. 2024Self-rationalization improves LLM as a fine-grained judgecited by 27
  23. 2023LLMEval: A Preliminary Study on How to Evaluate Large Language Modelscited by 23
  24. 2023INSTRUCTSCORE: Explainable Text Generation Evaluation with Finegrained Feedbackcited by 22
  25. 2025TRACT: Regression-Aware Fine-tuning Meets Chain-of-Thought Reasoning for LLM-as-a-Judgecited by 17
  26. 2024Reconfidencing LLMs from the Grouping Loss Perspectivecited by 11
  27. 2024The Comparative Trap: Pairwise Comparisons Amplifies Biased Preferences of LLM Evaluatorscited by 11
  28. 2025Understanding and Mitigating Bias Inheritance in LLM-based Data Augmentation on Downstream Taskscited by 9
  29. 2023JADE: A Linguistics-based Safety Evaluation Platform for Large Language Modelscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 4.

6 papers.

  1. 2026Hidden Measurement Error in LLM Pipelines Distorts Annotation, Evaluation, and Benchmarkingcited by 10
  2. 2026Efficient Inference for Noisy LLM-as-a-Judge Evaluationcited by 7
  3. 2026Calibrate, Don't Curate: Label-Efficient Estimation from Noisy LLM Judgescited by 5
  4. 2025EmeraldMind: A Knowledge Graph-Augmented Framework for Greenwashing Detectioncited by 0
  5. 2025Learning to Reason in LLMs by Expectation Maximizationcited by 0
  6. 2026Share the Judge, Learn the Deferral: Where Specialization Helps LLM Evaluationcited by 0