# The Importance of Multimodal Emotion Conditioning and Affect Consistency for Embodied Conversational Agents

arXiv: https://arxiv.org/abs/2309.15311 · HTML: https://mlsanity.com/paper/2309.15311/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 67 references that are in the map.
- First posted on arXiv 26 Sep 2023.

## Builds on (most cited first, 16 of 16)

- [Speech Gesture Generation from the Trimodal Context of Text, Audio, and Speaker Identity](https://mlsanity.com/paper/2009.02119/) (2020)
- [Learning Individual Styles of Conversational Gesture](https://mlsanity.com/paper/1906.04160/) (2019)
- [Emotional Chatting Machine: Emotional Conversation Generation with Internal and External Memory](https://mlsanity.com/paper/1704.01074/) (2017)
- [Robots Learn Social Skills: End-to-End Learning of Co-Speech Gesture Generation for Humanoid Robots](https://mlsanity.com/paper/1810.12541/) (2018)
- [Text2Gestures: A Transformer-Based Network for Generating Emotive Body Gestures for Virtual Agents](https://mlsanity.com/paper/2101.11101/) (2021)
- [Learning Speech-driven 3D Conversational Gestures from Video](https://mlsanity.com/paper/2102.06837/) (2021)
- [Speech2AffectiveGestures: Synthesizing Co-Speech Gestures with Generative Adversarial Affective Expression Learning](https://mlsanity.com/paper/2108.00262/) (2021)
- [Affect-Driven Dialog Generation](https://mlsanity.com/paper/1904.02793/) (2019)
- [Controllable Emotion Transfer For End-to-End Speech Synthesis](https://mlsanity.com/paper/2011.08679/) (2020)
- [A Review of Evaluation Practices of Gesture Generation in Embodied Conversational Agents](https://mlsanity.com/paper/2101.03769/) (2021)
- [An End-to-End Conversational Style Matching Agent](https://mlsanity.com/paper/1904.02760/) (2019)
- [Understanding the Predictability of Gesture Parameters from Speech and their Perceptual Importance](https://mlsanity.com/paper/2010.00995/) (2020)
- [Domain Authoring Assistant for Intelligent Virtual Agents](https://mlsanity.com/paper/1904.03266/) (2019)
- [EVA: Generating Emotional Behavior of Virtual Agents using Expressive Features of Gait and Gaze](https://mlsanity.com/paper/1907.02102/) (2019)
- [A Framework for Integrating Gesture Generation Models into Interactive Conversational Agents](https://mlsanity.com/paper/2102.12302/) (2021)
- [Transfer Learning from Monolingual ASR to Transcription-free Cross-lingual Voice Conversion](https://mlsanity.com/paper/2009.14668/) (2020)

## Built on by (most cited first, 4 of 4)

- [From Words to Worlds: Transforming One-line Prompt into Immersive Multi-modal Digital Stories with Communicative LLM Agent](https://mlsanity.com/paper/2406.10478/) (2024)
- [SpeechRole: A Large-Scale Dataset and Benchmark for Evaluating Speech Role-Playing Agents](https://mlsanity.com/paper/2508.02013/) (2025)
- [On the Equivalency, Substitutability, and Flexibility of Synthetic Data](https://mlsanity.com/paper/2403.16244/) (2024)
- [Beyond Global Emotion: Fine-Grained Emotional Speech Synthesis with Dynamic Word-Level Modulation](https://mlsanity.com/paper/2509.20378/) (2025)

## Abstract (arXiv)

Previous studies regarding the perception of emotions for embodied virtual agents have shown the effectiveness of using virtual characters in conveying emotions through interactions with humans. However, creating an autonomous embodied conversational agent with expressive behaviors presents two major challenges. The first challenge is the difficulty of synthesizing the conversational behaviors for each modality that are as expressive as real human behaviors. The second challenge is that the affects are modeled independently, which makes it difficult to generate multimodal responses with consistent emotions across all modalities. In this work, we propose a conceptual framework, ACTOR (Affect-Consistent mulTimodal behaviOR generation), that aims to increase the perception of affects by generating multimodal behaviors conditioned on a consistent driving affect. We have conducted a user study with 199 participants to assess how the average person judges the affects perceived from multimodal behaviors that are consistent and inconsistent with respect to a driving affect. The result shows that among all model conditions, our affect-consistent framework receives the highest Likert scores for the perception of driving affects. Our statistical analysis suggests that making a modality affect-inconsistent significantly decreases the perception of driving affects. We also observe that multimodal behaviors conditioned on consistent affects are more expressive compared to behaviors with inconsistent affects. Therefore, we conclude that multimodal emotion conditioning and affect consistency are vital to enhancing the perception of affects for embodied conversational agents.