# The Social Gaze of LLMs: A Literature Review of Multimodal Approaches to Human Behavior Understanding

arXiv: https://arxiv.org/abs/2510.23947 · HTML: https://mlsanity.com/paper/2510.23947/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Oct 2025.

## Built on by (most cited first, 1 of 1)

- [H2HMem: A Multimodal Memory Benchmark for Agents in Human-Human Interactions](https://mlsanity.com/paper/2606.09461/) (2026)

## Abstract (arXiv)

LLM-powered multimodal systems are increasingly used to interpret human behavior, yet how researchers apply the models' 'social competence' remains poorly understood. This paper presents a systematic literature review of 176 publications across different application domains (e.g., healthcare, education, and entertainment). Using a four-dimensional coding framework (application, technical, evaluative, and ethical), we find (1) frequent use of pattern recognition and information extraction from multimodal sources, but limited support for adaptive, interactive reasoning; (2) a dominant 'modality-to-text' pipeline that privileges language over rich audiovisual cues, striping away nuanced social cues; (3) evaluation practices reliant on static benchmarks, with socially grounded, human-centered assessments rare; and (4) Ethical discussions focused mainly on legal and rights-related risks (e.g., privacy), leaving societal risks (e.g., deception) overlooked--or at best acknowledged but left unaddressed. We outline a research agenda for evaluating socially competent, ethically informed, and interaction-aware multi-modal systems.