Towards Trust Calibration in Socially Interactive Agents: Investigating Gendered Multimodal Behaviors Generation with LLMs
On the map: LLM agents and tool use → Social simulation and multi-agent debate
Abstract (arXiv)
As Socially Interactive Agents (SIAs) become increasingly integrated into daily life, the ability to calibrate user trust to an agent's actual capabilities would help ensure appropriate usage of these agents. In this paper, we explore the capacity of Large Language Models (LLMs) to generate multimodal behaviors (verbal, vocal, gestural, and facial expression modalities) that reflect varying levels of ability and benevolence, two key dimensions of trustworthiness. We propose a novel method for automatically generating behaviors aligned with specific levels of these traits, a first step towards enabling nuanced and trust-calibrated interactions. By analyzing a large dataset of multimodal transcripts generated by LLMs, we demonstrate that GPT-5.4 is able to produce coherent behavior across different modalities (text, intonation, facial expression, and gesture). Using Random Forest feature importance analysis, we show that the generated behaviors align with theoretical expectations for ability and benevolence. However, we also find that when gender is specified in the prompt, LLMs tend to reproduce societal gender stereotypes, associating male agents' behaviors with high ability and female agents' behaviors with high benevolence. To validate our approach, we conducted a user study on Prolific using a within-subjects design. Participants perceived different levels of ability and benevolence in the generated behaviors align with the intended instructions.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Social simulation and multi-agent debate, one of 16 directions in LLM agents and tool use. Builds on 11 of its 49 references that are in the map. First posted on arXiv 19 May 2026.
Builds on
11 of 49 references.
Closest ancestors · 1
Same direction, newest first
Other references · 10
- 2021BABEL: Bodies, Action and Behavior with English Labelscited by 261
- 2024Motion-Agent: A Conversational Framework for Human Motion Generation with LLMscited by 46
- 2016DAiSEE: Towards User Engagement Recognition in the Wildcited by 39
- 2025Optimizing Temperature for Language Models with Multi-Sample Inferencecited by 33
- 2024EmoFace: Audio-driven Emotional 3D Face Animationcited by 9
- 2025Social Agent: Mastering Dyadic Nonverbal Behavior Generation via Conversational LLM Agentscited by 8
- 2016Controlling False Discoveries During Interactive Data Explorationcited by 5
- 2023Learning Through AI-Clones: Enhancing Self-Perception and Presentation Performancecited by 4
- 2025Speaking Beyond Language: A Large-Scale Multimodal Dataset for Learning Nonverbal Cues from Video-Grounded Dialoguescited by 4
- 2024Value Alignment and Trust in Human-Robot Interaction: Insights from Simulation and User Studycited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Evaluating LLM Alignment With Human Trust Models
- 2025A closer look at how large language models trust humans: patterns and biases
- 2026Do Personality-Tuned LLMs Make Better Social Agents?
- 2024Can Large Language Model Agents Simulate Human Trust Behavior?
- 2025Value-Based Large Language Model Agent Simulation for Mutual Evaluation of Trust and Interpersonal Closeness
- 2025The Social Laboratory: A Psychometric Framework for Multi-Agent LLM Evaluation
- 2024LLM Agents in Interaction: Measuring Personality Consistency and Linguistic Alignment in Interacting Populations of Large Language Models
- 2025Humanizing LLMs: A Survey of Psychological Measurements with Tools, Datasets, and Human-Agent Applications