# Are they lovers or friends? Evaluating LLMs' Social Reasoning in English and Korean Dialogues

arXiv: https://arxiv.org/abs/2510.19028 · HTML: https://mlsanity.com/paper/2510.19028/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 13 of its 29 references that are in the map.
- First posted on arXiv 21 Oct 2025.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 13 of 13)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation](https://mlsanity.com/paper/2104.14478/) (2021)
- [Dialogue-Based Relation Extraction](https://mlsanity.com/paper/2004.08056/) (2020)
- [EXAONE 4.0: Unified Large Language Models Integrating Non-reasoning and Reasoning Modes](https://mlsanity.com/paper/2507.11407/) (2025)
- [Hypothesis-Driven Theory-of-Mind Reasoning for Large Language Models](https://mlsanity.com/paper/2502.11881/) (2025)
- [SocialDial: A Benchmark for Socially-Aware Dialogue Systems](https://mlsanity.com/paper/2304.12026/) (2023)
- [Kanana: Compute-efficient Bilingual Language Models](https://mlsanity.com/paper/2502.18934/) (2025)
- [PeerGPT: Probing the Roles of LLM-based Peer Agents as Team Moderators and Participants in Children's Collaborative Learning](https://mlsanity.com/paper/2403.14227/) (2024)
- [DiPlomat: A Dialogue Dataset for Situated Pragmatic Reasoning](https://mlsanity.com/paper/2306.09030/) (2023)
- [DDRel: A New Dataset for Interpersonal Relation Classification in Dyadic Dialogues](https://mlsanity.com/paper/2012.02553/) (2020)
- [FineD-Eval: Fine-grained Automatic Dialogue-Level Evaluation](https://mlsanity.com/paper/2210.13832/) (2022)
- [Your spouse needs professional help: Determining the Contextual Appropriateness of Messages through Modeling Social Relationships](https://mlsanity.com/paper/2307.02763/) (2023)

## Built on by (most cited first, 3 of 3)

- [World in a Frame: Understanding Culture Mixing as a New Challenge for Vision-Language Models](https://mlsanity.com/paper/2511.22787/) (2025)
- [Pseudo-Deliberation in Language Models: When Reasoning Fails to Align Values and Actions](https://mlsanity.com/paper/2605.09893/) (2026)
- [FriendBench: Benchmarking Dyadic Familiarity Inference in Humans and Multimodal Large Language Models](https://mlsanity.com/paper/2607.29602/) (2026)

## Abstract (arXiv)

As LLMs are increasingly deployed in real-world interactions, their social reasoning in interpersonal communication becomes critical. To explore their capabilities, we introduce SCRIPTS, a 1.1k-dialogue dataset in English and Korean, sourced from movie scripts and propose a social reasoning task based on SCRIPTS that evaluates the capacity of LLMs to infer the social relationships (e.g., friends, lovers) between speakers in each dialogue. Evaluating nine models on our task, current LLMs achieve around 75--80% on the English dataset and 58--69% in Korean, and models predict an Unlikely relationship in 10--25% of responses in both languages. Furthermore, we find that thinking models and chain-of-thought prompting provide minimal benefits for social reasoning and occasionally amplify social biases. In sum, there are significant limitations in current LLMs' social reasoning capabilities, especially for Korean, highlighting the need for efforts to develop socially-aware LLMs across languages.