mlsanityData to 7 Oct 2026

aipsy-judge: A Specialized, Psychologist-Corrected Local Judge for the Psychological Safety of Conversational AI

arXiv 2608.24899 ↗First posted 13 Jul 2026cs.HC cs.AI cs.CYcited by 0

Not on the map. Why?

Abstract (arXiv)

The standard recipe for LLM-as-judge -- pick a frontier model, or average several -- is actively unsafe for grading the psychological safety of conversational AI. Using aipsy-bench, an open frozen safety instrument, we run a fully-crossed competence study: three frontier models (gpt-5.4-mini, claude-sonnet-4-6, gemini-2.5-flash) serve as both generators and judges of 3,000 mental-health, companion, and coaching messages against a psychologist's ratings. The disagreement is not noise: it is structured, concentrated on the safety-critical metrics, and one judge (Gemini) is an outlier -- the most lenient, carrying a +0.99 self-preference premium, flagging far fewer tail failures, and scoring a means-in-hand self-harm response "exemplary." Inter-judge agreement on empathy, where sycophancy hides, is the lowest in the battery (alpha 0.24). One axis stands apart: the binary crisis-detection flag is the one safety-critical signal judges agree on (alpha 0.80), erring toward over-flagging, the safe direction for a triage screen. Equal-weight averaging, the canonical fix, blends that leniency and tail-blindness into the safety score. Off-the-shelf open-weight judges are worse for a dispositional, not capability, reason -- and disposition is fine-tunable. We therefore distill a per-metric, psychologist-corrected target into a small, frozen, local model, aipsy-judge-1.0, an Apache-2.0 fine-tune of Gemma-4-26B-A4B. aipsy-judge-1.0 tracks the corrected target better than its base on the composite (ICC 0.64 to 0.75) and crisis detection (kappa 0.65 to 0.82), catches 92% of crises with a false-positive lean, and grades more faithfully than any single frontier judge, while every transcript stays on the machine. These are directional readings against a single-expert-informed target, not validated multi-rater agreement. A safety grader that shares a vendor's post-training shares its blind spots.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 22 of its 27 references that are in the map. First posted on arXiv 13 Jul 2026.

Builds on

22 of 27 references.

References · 22

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  4. 2024A Survey on LLM-as-a-Judgecited by 1,384
  5. 2019Class-Balanced Loss Based on Effective Number of Samplescited by 1,239
  6. 2023Towards Understanding Sycophancy in Language Modelscited by 988
  7. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 651
  8. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 558
  9. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 431
  10. 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 354
  11. 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 351
  12. 2024Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Modelscited by 301
10 more
  1. 2024Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judgecited by 249
  2. 2024Self-Preference Bias in LLM-as-a-Judgecited by 248
  3. 2025No Free Labels: Limitations of LLM-as-a-Judge Without Human Groundingcited by 50
  4. 2025Neither Valid nor Reliable? Investigating the Use of LLMs as Judgescited by 37
  5. 2025Safer or Luckier? LLMs as Safety Evaluators Are Not Robust to Artifactscited by 25
  6. 2025Training language models to be warm and empathetic makes them less reliable and more sycophanticcited by 19
  7. 2025When Can We Trust LLMs in Mental Health? Large-Scale Benchmarks for Reliable LLM Evaluationcited by 19
  8. 2025Distill Not Only Data but Also Rewards: Can Smaller Language Models Surpass Larger Ones?cited by 12
  9. 2026Using LLM-as-a-Judge/Jury to Advance Scalable, Clinically-Validated Safety Evaluations of Model Responses to Users Demonstrating Psychosiscited by 3
  10. 2026Agreement Metrics for LLM-as-Judge Evaluation: What to Report and Whycited by 3

Built on by

Not cited yet.