HealthBench-Psych: A Mental Health Subset of OpenAI's HealthBench
Not on the map. Why?
Abstract (arXiv)
General-purpose health benchmarks increasingly anchor claims about LLM medical performance, but they are not always resolved by clinical specialty, making domain-specific performance hard to isolate. Mental health is of acute public-health concern as millions of people turn to LLMs for psychological support, and most existing evaluations are bespoke academic benchmarks that are difficult to integrate into developer workflows. We introduce HealthBench-Psych and HealthBench-Psych-Hard. We screened HealthBench's 5,000 physician-rubric conversations for mental-health relevance with a transparent LLM-applied rubric, then validated the subset through two rounds of blinded clinician review with concealed known-exclude controls, yielding 610 conversations (12.2% of the corpus). Evaluating 20 frontier and open models under a cross-vendor panel of three LLM judges, we find a statistically tied frontier cluster, measurable refusal behavior in two models, and near-identical rankings across judges (τ ≥ 0.92). We release the subset, pipeline, model responses, grades, and analysis code as a reusable resource.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 3 of its 17 references that are in the map. First posted on arXiv 25 Aug 2026.
Builds on
3 of 17 references.
References · 3
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
- 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 686
- 2025HealthBench: Evaluating Large Language Models Towards Improved Human Healthcited by 429
Built on by
Not cited yet.