# SPLIT: Cross-Lingual Empathy and Cultural Grounding in English and Ukrainian LLM Responses

arXiv: https://arxiv.org/abs/2607.02049 · HTML: https://mlsanity.com/paper/2607.02049/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 37 of its 62 references that are in the map.
- First posted on arXiv 2 Jul 2026.

## Builds on (most cited first, 30 of 37)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [The State and Fate of Linguistic Diversity and Inclusion in the NLP World](https://mlsanity.com/paper/2004.09095/) (2020)
- [LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks](https://mlsanity.com/paper/2406.18403/) (2024)
- [Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts](https://mlsanity.com/paper/2408.15664/) (2024)
- [Cultural Alignment in Large Language Models: An Explanatory Analysis Based on Hofstede's Cultural Dimensions](https://mlsanity.com/paper/2309.12342/) (2023)
- [An Empirical Study of LLM-as-a-Judge for LLM Evaluation: Fine-tuned Judge Model is not a General Substitute for GPT-4](https://mlsanity.com/paper/2403.02839/) (2024)
- [EQ-Bench: An Emotional Intelligence Benchmark for Large Language Models](https://mlsanity.com/paper/2312.06281/) (2023)
- [Towards Facilitating Empathic Conversations in Online Mental Health Support: A Reinforcement Learning Approach](https://mlsanity.com/paper/2101.07714/) (2021)
- [Evaluating Scoring Bias in LLM-as-a-Judge](https://mlsanity.com/paper/2506.22316/) (2025)
- [Multilingual != Multicultural: Evaluating Gaps Between Multilingual Capabilities and Cultural Alignment in LLMs](https://mlsanity.com/paper/2502.16534/) (2025)
- [Putting GPT-4o to the Sword: A Comprehensive Evaluation of Language, Vision, Speech, and Multimodal Proficiency](https://mlsanity.com/paper/2407.09519/) (2024)
- [How Reliable is Multilingual LLM-as-a-Judge?](https://mlsanity.com/paper/2505.12201/) (2025)
- [Judge's Verdict: A Comprehensive Analysis of LLM Judge Capability Through Human Agreement](https://mlsanity.com/paper/2510.09738/) (2025)
- [Paths Not Taken: Understanding and Mending the Multilingual Factual Recall Pipeline](https://mlsanity.com/paper/2505.20546/) (2025)
- [MuBench: Assessment of Multilingual Capabilities of Large Language Models Across 61 Languages](https://mlsanity.com/paper/2506.19468/) (2025)
- [Self-Preference Bias in Rubric-Based Evaluation of Large Language Models](https://mlsanity.com/paper/2604.06996/) (2026)
- [Multi-dimensional Evaluation of Empathetic Dialog Responses](https://mlsanity.com/paper/2402.11409/) (2024)
- [Are LLMs Empathetic to All? Investigating the Influence of Multi-Demographic Personas on a Model's Empathy](https://mlsanity.com/paper/2510.10328/) (2025)
- [JudgeSense: A Benchmark for Prompt Sensitivity in LLM-as-a-Judge Systems](https://mlsanity.com/paper/2604.23478/) (2026)
- [Judging the Judges: A Systematic Evaluation of Bias Mitigation Strategies in LLM-as-a-Judge Pipelines](https://mlsanity.com/paper/2604.23178/) (2026)
- [Improbable Bigrams Expose Vulnerabilities of Incomplete Tokens in Byte-Level Tokenizers](https://mlsanity.com/paper/2410.23684/) (2024)
- [Understanding and Enhancing the Planning Capability of Language Models via Multi-Token Prediction](https://mlsanity.com/paper/2509.23186/) (2025)
- [Empathic Grounding: Explorations using Multimodal Interaction and Large Language Models with Conversational Agents](https://mlsanity.com/paper/2407.01824/) (2024)
- [Evaluating Knowledge-based Cross-lingual Inconsistency in Large Language Models](https://mlsanity.com/paper/2407.01358/) (2024)
- [The Tokenizer Tax Across 25 European Languages: Domain Invariance, Cross-Lingual Few-Shot Effects, and the Ukrainian Penalty](https://mlsanity.com/paper/2605.24718/) (2026)
- [IndicParam: Benchmark to evaluate LLMs on low-resource Indic Languages](https://mlsanity.com/paper/2512.00333/) (2025)

## Abstract (arXiv)

Large Language Models are increasingly deployed in emotional-support contexts and crisis-related situations. Nevertheless, their cross-lingual abilities in these circumstances remain underexplored. Existing benchmarks emphasize multilingual performance but rarely examine crisis-related empathy and cultural grounding in low-to-mid-resource languages. We introduce SPLIT, a 500-prompt benchmark designed to evaluate LLM consistency in generating emotionally grounded responses across five categories: Stress, Panic, Loneliness, Internal Displacement, and Tension. We evaluate three technically diverse LLMs across three dimensions: Empathetic Accuracy, Linguistic Naturalness, and Contextual & Cultural Grounding. The framework aims to assess and compare the quality of LLM responses in both English and Ukrainian languages, as well as to explore the reliability of the LLM-as-a-jury paradigm. Our findings reveal that Gemini-2.5-Flash and LLaMA-3.3-70B-Instruct degrade when transitioning to Ukrainian, while DeepSeek-V3 remains comparatively stable within our benchmark. We additionally find that human and AI evaluators agree weakly on empathy and naturalness but diverge on cultural grounding. We further argue that producing Ukrainian text is not equivalent to producing Ukrainian emotional support. Our findings may assist in the future development of more culturally tailored benchmark designs, as well as encourage a stronger emphasis on human-centered evaluation.