mlsanityData to 7 Oct 2026

When Trust Meets Truth: Trust-Truth Separability in LLM-as-Judge

arXiv 2608.21097 ↗First posted 21 Aug 2026cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

LLM-as-Judge systems can produce multi-dimensional evaluations, such as trustworthiness, reliability, and factuality, and these outputs are often interpreted as independent evidence. We test this assumption for a common pair of judgments: trust scoring and binary truth classification. On correctness-controlled QA, LLM judges align trust scores with truth verdicts more tightly than human behavioral reference, suggesting weaker separations between trust and truth judgment. We then apply stress tests by changing only source cues of identical QA between Human and AI. Source attribution shifts not only trust scores but also truth verdicts and logit-derived correct-side probabilities. Results show that current LLM-as-Judge protocols should not treat trust scores as independent evidence for truth judgments.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 19 of its 29 references that are in the map. First posted on arXiv 21 Aug 2026.

Builds on

19 of 29 references.

References · 19

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  3. 2024A Survey on LLM-as-a-Judgecited by 1,384
  4. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 480
  5. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 451
  6. 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 354
  7. 2024Humans or LLMs as the Judge? A Study on Judgement Biasescited by 314
  8. 2024Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judgecited by 249
  9. 2024Self-Preference Bias in LLM-as-a-Judgecited by 248
  10. 2022ASQA: Factoid Questions Meet Long-Form Answerscited by 231
  11. 2019A Question-Entailment Approach to Question Answeringcited by 133
  12. 2024Can You Trust LLM Judgments? Reliability of LLM-as-a-Judgecited by 52
7 more
  1. 2022Designing for Responsible Trust in AI Systems: A Communication Perspectivecited by 48
  2. 2025Evaluating Scoring Bias in LLM-as-a-Judgecited by 42
  3. 2025LLMs Cannot Reliably Judge (Yet?): A Comprehensive Assessment on the Robustness of LLM-as-a-Judgecited by 33
  4. 2025Analyzing Uncertainty of LLM-as-a-Judge: Interval Evaluations with Conformal Predictioncited by 26
  5. 2025TrustJudge: Inconsistencies of LLM-as-a-Judge and How to Alleviate Themcited by 20
  6. 2025The Silent Judge: Unacknowledged Shortcut Bias in LLM-as-a-Judgecited by 13
  7. 2026Label Effects: Shared Heuristic Reliance in Trust Assessment by Humans and LLM-as-a-Judgecited by 3

Built on by

Not cited yet.