mlsanityData to 7 Oct 2026

Can You Trust LLM Judgments? Reliability of LLM-as-a-Judge

arXiv 2412.12509 ↗First posted 17 Dec 2024cs.CLcited by 52

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) have become increasingly powerful and ubiquitous, but their stochastic nature poses challenges to the reliability of their outputs. While deterministic settings can improve consistency, they do not guarantee reliability, as a single sample from the model's probability distribution can still be misleading. Building upon the concept of LLM-as-a-judge, we introduce a novel framework for rigorously evaluating the reliability of LLM judgments, leveraging McDonald's omega. We evaluate the reliability of LLMs when judging the outputs of other LLMs on standard single-turn and multi-turn benchmarks, simultaneously investigating the impact of temperature on reliability. By analyzing these results, we demonstrate the limitations of fixed randomness and the importance of considering multiple samples, which we show has significant implications for downstream applications. Our findings highlight the need for a nuanced understanding of LLM reliability and the potential risks associated with over-reliance on single-shot evaluations. This work provides a crucial step towards building more trustworthy and reliable LLM-based systems and applications.

Key facts. Cited by 52 papers in the map (data through 7 Oct 2026). First posted on arXiv 17 Dec 2024.

Builds on

0 of 0 references.

No references here yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 16; 2026: 36.

52 papers.

Sort:

By area:

  1. 2025Analyzing Uncertainty of LLM-as-a-Judge: Interval Evaluations with Conformal Predictioncited by 26
  2. 2026Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Biascited by 19
  3. 2025An Empirical Study of LLM-as-a-Judge: How Design Choices Impact Evaluation Reliabilitycited by 17
  4. 2026Diagnosing the Reliability of LLM-as-a-Judge via Item Response Theorycited by 13
  5. 2026FIRE-Bench: Evaluating AI Agents on the Rediscovery of Scientific Insightscited by 13
  6. 2025Do not Abstain! Identify and Solve the Uncertaintycited by 11
  7. 2025RecGPT Technical Reportcited by 11
  8. 2026Benchmarks Saturate When The Model Gets Smarter Than The Judgecited by 10
  9. 2025SAC-Opt: Semantic Anchors for Iterative Correction in Optimization Modelingcited by 8
  10. 2026Efficient Inference for Noisy LLM-as-a-Judge Evaluationcited by 7
  11. 2025Reasoning LLMs are Wandering Solution Explorerscited by 6
  12. 2026ChartEditBench: Evaluating Grounded Multi-Turn Chart Editing in Multimodal Language Modelscited by 6
38 more
  1. 2026LOGIGEN: Logic-Driven Generation of Verifiable Agentic Taskscited by 5
  2. 2026ContextEcho: A Benchmark for Persona Drift in Long Agentic-Coding Sessionscited by 5
  3. 2025Beyond Words and Pixels: A Benchmark for Implicit World Knowledge Reasoning in Generative Modelscited by 4
  4. 2026MTMCS-Bench: Evaluating Contextual Safety of Multimodal Large Language Models in Multi-Turn Dialoguescited by 4
  5. 2026CyclicJudge: Mitigating Judge Bias Efficiently in LLM-based Evaluationcited by 4
  6. 2026Ontology-Constrained Neural Reasoning in Enterprise Agentic Systems: A Neurosymbolic Architecture for Domain-Grounded AI Agentscited by 3
  7. 2026Label Effects: Shared Heuristic Reliance in Trust Assessment by Humans and LLM-as-a-Judgecited by 3
  8. 2026ComplexConstraints and Beyond: Expert Rubrics for RLVRcited by 3
  9. 2025RephQA: Evaluating Readability of Large Language Models in Public Health Question Answeringcited by 2
  10. 2025Validating the Effectiveness of a Large Language Model-based Approach for Identifying Children's Development across Various Free Play Settings in Kindergartencited by 1
  11. 2025IntelliProof: An Argumentation Network-based Conversational Helper for Organized Reflectioncited by 1
  12. 2025Safe for Whom? Rethinking How We Evaluate the Safety of LLMs for Real Userscited by 1
  13. 2026Trustworthy Agent Network: Trust in Agent Networks Must Be Baked In, Not Bolted Oncited by 1
  14. 2026A Multi-Agent LLM Framework for Rating the Quality of Surgical Feedbackcited by 1
  15. 2026VIABLE: A Visually Impaired Assistance Benchmark for VLM-as-a-Judge Evaluationcited by 1
  16. 2026Escaping the Self-Confirmation Trap: An Execute-Distill-Verify Paradigm for Agentic Experience Learningcited by 1
  17. 2026Democratic ICAI: Debating Our Way to Steering Principles from Preferencescited by 1
  18. 2026Judge, Retrieve, or Abstain: Uncertainty-Guarded LLM Judging with Provable Risk Guaranteescited by 1
  19. 2025Meta-Evaluating Local LLMs: Rethinking Performance Metrics for Serious Gamescited by 0
  20. 2025IDEAlign: Comparing Ideas of Large Language Models to Domain Expertcited by 0
  21. 2025What Work is AI Actually Doing? Uncovering the Drivers of Generative AI Adoptioncited by 0
  22. 2025RAGVUE: A Diagnostic View for Explainable and Automated Evaluation of Retrieval-Augmented Generationcited by 0
  23. 2025Gaming the Answer Matcher: Examining the Impact of Text Manipulation on Automated Judgmentcited by 0
  24. 2026METIS: Mentoring Engine for Thoughtful Inquiry & Solutionscited by 0
  25. 2026Same Input, Different Scores: A Multi Model Study on the Inconsistency of LLM Judgecited by 0
  26. 2026The Hrunting of AI: Where and How to Improve English Dialectal Fairnesscited by 0
  27. 2026LLM-as-Judge for Semantic Judging of Powerline Segmentation in UAV Inspectioncited by 0
  28. 2026Large Language Models Outperform Humans in Fraud Detection and Resistance to Motivated Investor Pressurecited by 0
  29. 2026A Systematic Investigation of RL-Jailbreaking in LLMscited by 0
  30. 2026A Communication-Theoretic Framework for LLM Agents: Cost-Aware Adaptive Reliabilitycited by 0
  31. 2026Are We Evaluating Knowledge or Phrasing? Mitigating MCQA Sensitivity with ParaEvalcited by 0
  32. 2026Intelligence Is Not the Bottleneck: Validating an LLM First-Pass Manuscript Score Against Peer-Review Outcomescited by 0
  33. 2026Reexamining zero-shot summarization: Empirical investigation of trustworthiness of LLM-summarizerscited by 0
  34. 2026Codifying the Judge: Scalable Evaluation via Program Distillationcited by 0
  35. 2026ASSERT: A Measurement Pipeline for GenAI Auditscited by 0
  36. 2026When Trust Meets Truth: Trust-Truth Separability in LLM-as-Judgecited by 0
  37. 2026Multilingual in Name Only? Cultural and Linguistic Weaknesses of LLMs in Urducited by 0
  38. 2026Three Ways Classical Test Theory Can Mislead About LLM Judgescited by 0

50 most cited of 52 All 52 (JSON)