# Reading the Mood Behind Words: Integrating Prosody-Derived Emotional Context into Socially Responsive VR Agents

arXiv: https://arxiv.org/abs/2603.09324 · HTML: https://mlsanity.com/paper/2603.09324/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 33 references that are in the map.
- First posted on arXiv 10 Mar 2026.

## Builds on (most cited first, 12 of 12)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [SUPERB: Speech processing Universal PERformance Benchmark](https://mlsanity.com/paper/2105.01051/) (2021)
- [Emotional Speech-Driven Animation with Content-Emotion Disentanglement](https://mlsanity.com/paper/2306.08990/) (2023)
- [Artificial social influence via human-embodied AI agent interaction in immersive virtual reality (VR): Effects of similarity-matching during health conversations](https://mlsanity.com/paper/2406.05486/) (2024)
- [ELLMA-T: an Embodied LLM-agent for Supporting English Language Learning in Social VR](https://mlsanity.com/paper/2410.02406/) (2024)
- [It's not what you said, it's how you said it: discriminative perception of speech as a multichannel communication system](https://mlsanity.com/paper/2105.00260/) (2021)
- [Prosody in Cascade and Direct Speech-to-Text Translation: a case study on Korean Wh-Phrases](https://mlsanity.com/paper/2402.00632/) (2024)
- [Evaluating Emotion Recognition in Spoken Language Models on Emotionally Incongruent Speech](https://mlsanity.com/paper/2510.25054/) (2025)
- [Quantifying the perceptual value of lexical and non-lexical channels in speech](https://mlsanity.com/paper/2307.03534/) (2023)
- [Empathetic Conversational Agents: Utilizing Neural and Physiological Signals for Enhanced Empathetic Interactions](https://mlsanity.com/paper/2501.08393/) (2025)
- [Designing VR Simulation System for Clinical Communication Training with LLMs-Based Embodied Conversational Agents](https://mlsanity.com/paper/2503.01767/) (2025)

## Abstract (arXiv)

In VR interactions with embodied conversational agents, users' emotional intent is often conveyed more by how something is said than by what is said. However, most VR agent pipelines rely on speech-to-text processing, discarding prosodic cues and often producing emotionally incongruent responses despite correct semantics. We propose an emotion-context-aware VR interaction pipeline that treats vocal emotion as explicit dialogue context in an LLM-based conversational agent. A real-time speech emotion recognition model infers users' emotional states from prosody, and the resulting emotion labels are injected into the agent's dialogue context to shape response tone and style. Results from a within-subjects VR study (N=30) show significant improvements in dialogue quality, naturalness, engagement, rapport, and human-likeness, with 93.3% of participants preferring the emotion-aware agent.