# Takeaways from Applying LLM Capabilities to Multiple Conversational Avatars in a VR Pilot Study

arXiv: https://arxiv.org/abs/2501.00168 · HTML: https://mlsanity.com/paper/2501.00168/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Dec 2024.

## Built on by (most cited first, 7 of 7)

- [Mitigating Response Delays in Free-Form Conversations with LLM-powered Intelligent Virtual Agents](https://mlsanity.com/paper/2507.22352/) (2025)
- [Integrating Personality into Digital Humans: A Review of LLM-Driven Approaches for Virtual Reality](https://mlsanity.com/paper/2503.16457/) (2025)
- [How LLMs are Shaping the Future of Virtual Reality](https://mlsanity.com/paper/2508.00737/) (2025)
- [Behavioral and Symbolic Fillers as Delay Mitigation for Embodied Conversational Agents in Virtual Reality](https://mlsanity.com/paper/2508.11781/) (2025)
- [CLiVR: Conversational Learning System in Virtual Reality with AI-Powered Patients](https://mlsanity.com/paper/2510.19031/) (2025)
- [A Study of the Framework and Real-World Applications of Language Embedding for 3D Scene Understanding](https://mlsanity.com/paper/2508.05064/) (2025)
- [ELEVATE: Designing Human-Centered GenAI Virtual Tutors for Scalable and Inclusive Education](https://mlsanity.com/paper/2606.30662/) (2026)

## Abstract (arXiv)

We present a virtual reality (VR) environment featuring conversational avatars powered by a locally-deployed LLM, integrated with automatic speech recognition (ASR), text-to-speech (TTS), and lip-syncing. Through a pilot study, we explored the effects of three types of avatar status indicators during response generation. Our findings reveal design considerations for improving responsiveness and realism in LLM-driven conversational systems. We also detail two system architectures: one using an LLM-based state machine to control avatar behavior and another integrating retrieval-augmented generation (RAG) for context-grounded responses. Together, these contributions offer practical insights to guide future work in developing task-oriented conversational AI in VR environments.