# Dynamic Multimodal Expression Generation for LLM-Driven Pedagogical Agents: From User Experience Perspective

arXiv: https://arxiv.org/abs/2603.09536 · HTML: https://mlsanity.com/paper/2603.09536/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Educational agents and math problem solving, one of 16 directions in LLM agents and tool use.
- Builds on 7 of its 54 references that are in the map.
- First posted on arXiv 10 Mar 2026.

## Placed in

- LLM agents and tool use / [Educational agents and math problem solving](https://mlsanity.com/agents/educational-agents-and-math-problem-solving/)

## Builds on (most cited first, 7 of 7)

- [Mitigating Response Delays in Free-Form Conversations with LLM-powered Intelligent Virtual Agents](https://mlsanity.com/paper/2507.22352/) (2025)
- [An Aligned Rank Transform Procedure for Multifactor Contrast Tests](https://mlsanity.com/paper/2102.11824/) (2021)
- [LLMER: Crafting Interactive Extended Reality Worlds with JSON Data Generated by Large Language Models](https://mlsanity.com/paper/2502.02441/) (2025)
- [Explainable XR: Understanding User Behaviors of XR Environments using LLM-assisted Analytics Framework](https://mlsanity.com/paper/2501.13778/) (2025)
- [Exploring LLM-Powered Role and Action-Switching Pedagogical Agents for History Education in Virtual Reality](https://mlsanity.com/paper/2505.02699/) (2025)
- [Quantifying the Effects of Working in VR for One Week](https://mlsanity.com/paper/2206.03189/) (2022)
- [Enhancing Patient Acceptance of Robotic Ultrasound through Conversational Virtual Agent and Immersive Visualizations](https://mlsanity.com/paper/2502.10088/) (2025)

## Abstract (arXiv)

In virtual reality (VR) educational scenarios, Pedagogical agents (PAs) enhance immersive learning through realistic appearances and interactive behaviors. However, most existing PAs rely on static speech and simple gestures. This limitation reduces their ability to dynamically adapt to the semantic context of instructional content. As a result, interactions often lack naturalness and effectiveness in the teaching process. To address this challenge, this study proposes a large language model (LLM)-driven multimodal expression generation method that constructs semantically sensitive prompts to generate coordinated speech and gesture instructions, enabling dynamic alignment between instructional semantics and multimodal expressive behaviors. A VR-based PA prototype was developed and evaluated through user experience-oriented subjective experiments. Results indicate that dynamically generated multimodal expressions significantly enhance learners' perceived learning effectiveness, engagement, and intention to use, while effectively alleviating feelings of fatigue and boredom during the learning process. Furthermore, the combined dynamic expression of speech and gestures notably enhances learners' perceptions of human-likeness and social presence. The findings provide new insights and design guidelines for building more immersive and naturally expressive intelligent PAs.