# Language Shapes Mental Health Evaluations in Large Language Models

arXiv: https://arxiv.org/abs/2603.06910 · HTML: https://mlsanity.com/paper/2603.06910/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 65 references that are in the map.
- First posted on arXiv 6 Mar 2026.

## Builds on (most cited first, 13 of 13)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Large Language Models for Information Retrieval: A Survey](https://mlsanity.com/paper/2308.07107/) (2023)
- [Cross-Lingual Consistency of Factual Knowledge in Multilingual Language Models](https://mlsanity.com/paper/2310.10378/) (2023)
- [Human-AI Collaboration Enables More Empathic Conversations in Text-based Peer-to-Peer Mental Health Support](https://mlsanity.com/paper/2203.15144/) (2022)
- [Language Model Alignment in Multilingual Trolley Problems](https://mlsanity.com/paper/2407.02273/) (2024)
- [Social Bias Evaluation for Large Language Models Requires Prompt Variations](https://mlsanity.com/paper/2407.03129/) (2024)
- [This Land is {Your, My} Land: Evaluating Geopolitical Biases in Language Models](https://mlsanity.com/paper/2305.14610/) (2023)
- [Exploring Multilingual Concepts of Human Value in Large Language Models: Is Value Alignment Consistent, Transferable and Controllable across Languages?](https://mlsanity.com/paper/2402.18120/) (2024)
- [LLMs and Cultural Values: the Impact of Prompt Language and Explicit Cultural Framing](https://mlsanity.com/paper/2511.03980/) (2025)
- [Multilingual Retrieval Augmented Generation for Culturally-Sensitive Tasks: A Benchmark for Cross-lingual Robustness](https://mlsanity.com/paper/2410.01171/) (2024)
- [Aligning LLMs for Multilingual Consistency in Enterprise Applications](https://mlsanity.com/paper/2509.23659/) (2025)
- [What is Stigma Attributed to? A Theory-Grounded, Expert-Annotated Interview Corpus for Demystifying Mental-Health Stigma](https://mlsanity.com/paper/2505.12727/) (2025)
- [Fairness or Fluency? An Investigation into Language Bias of Pairwise LLM-as-a-Judge](https://mlsanity.com/paper/2601.13649/) (2026)

## Built on by (most cited first, 2 of 2)

- [EMPATH: A Multilingual Auditor-Judge Benchmark for Safety Evaluation of Emotional-Support Chatbots](https://mlsanity.com/paper/2606.30256/) (2026)
- [Language-model ratings of depression reflect the rater more than the patient](https://mlsanity.com/paper/2610.08501/) (2026)

## Abstract (arXiv)

Multilingual large language models (LLMs) are increasingly used in socially sensitive mental health contexts, including support chatbots, screening, and content moderation. This raises a reliability question: do semantically equivalent mental health inputs elicit comparable evaluations across languages, or systematic shifts consistent with language-associated social and cultural contexts? We examine this question in an English-Chinese setting with GPT-4o and Qwen3-32B using a two-level framework: construct-level evaluative orientation, measured by psychometric stigma instruments, and decision-level behavior, measured by binary stigma detection and four-class depression severity classification. Across instruments and models, Chinese prompts elicit higher stigma-related scores than English prompts. At the decision level, Chinese prompts reduce sensitivity to stigmatizing content and produce more conservative depression severity judgments, leading to more under-estimation errors. These findings show that prompt language can shift both evaluative orientation and downstream behavior in LLM-based mental health evaluation. They highlight the need to evaluate multilingual LLMs not only for aggregate performance, but also for whether they apply comparable evaluative standards across languages in socially sensitive domains.