# CUIfy the XR: An Open-Source Package to Embed LLM-powered Conversational Agents in XR

arXiv: https://arxiv.org/abs/2411.04671 · HTML: https://mlsanity.com/paper/2411.04671/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Social simulation and multi-agent debate, one of 16 directions in LLM agents and tool use.
- Builds on 11 of its 52 references that are in the map.
- First posted on arXiv 7 Nov 2024.

## Placed in

- LLM agents and tool use / [Social simulation and multi-agent debate](https://mlsanity.com/agents/social-simulation-and-multi-agent-debate/)

## Builds on (most cited first, 11 of 11)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledge](https://mlsanity.com/paper/2303.14070/) (2023)
- [(A)I Am Not a Lawyer, But...: Engaging Legal Experts towards Responsible LLM Policies for Legal Advice](https://mlsanity.com/paper/2402.01864/) (2024)
- [LLMR: Real-time Prompting of Interactive Worlds using Large Language Models](https://mlsanity.com/paper/2309.12276/) (2023)
- [Embedding Large Language Models into Extended Reality: Opportunities and Challenges for Inclusion, Engagement, and Privacy](https://mlsanity.com/paper/2402.03907/) (2024)
- [Automated Assessment of Encouragement and Warmth in Classrooms Leveraging Multimodal Emotional Features and ChatGPT](https://mlsanity.com/paper/2404.15310/) (2024)
- [Touch? Speech? or Touch and Speech? Investigating Multimodal Interaction for Visual Network Exploration and Analysis](https://mlsanity.com/paper/2004.14505/) (2020)
- [Evaluating Usability and Engagement of Large Language Models in Virtual Reality for Traditional Scottish Curling](https://mlsanity.com/paper/2408.09285/) (2024)
- [Voice2Action: Language Models as Agent for Efficient Real-Time Interaction in Virtual Reality](https://mlsanity.com/paper/2310.00092/) (2023)
- [Wrapped in Anansi's Web: Unweaving the Impacts of Generative-AI Personalization and VR Immersion in Oral Storytelling](https://mlsanity.com/paper/2409.16894/) (2024)

## Built on by (most cited first, 5 of 5)

- [Large language models for artificial general intelligence (AGI): A survey of foundational principles and approaches](https://mlsanity.com/paper/2501.03151/) (2025)
- [Exploring Context-aware and LLM-driven Locomotion for Immersive Virtual Reality](https://mlsanity.com/paper/2504.17331/) (2025)
- [A Review on Large Language Models for Visual Analytics](https://mlsanity.com/paper/2503.15176/) (2025)
- [Text2VR: Automated instruction Generation in Virtual Reality using Large language Models for Assembly Task](https://mlsanity.com/paper/2508.03699/) (2025)
- [Exploring User Acceptance and Concerns toward LLM-powered Conversational Agents in Immersive Extended Reality](https://mlsanity.com/paper/2512.15343/) (2025)

## Abstract (arXiv)

Recent developments in computer graphics, machine learning, and sensor technologies enable numerous opportunities for extended reality (XR) setups for everyday life, from skills training to entertainment. With large corporations offering affordable consumer-grade head-mounted displays (HMDs), XR will likely become pervasive, and HMDs will develop as personal devices like smartphones and tablets. However, having intelligent spaces and naturalistic interactions in XR is as important as technological advances so that users grow their engagement in virtual and augmented spaces. To this end, large language model (LLM)--powered non-player characters (NPCs) with speech-to-text (STT) and text-to-speech (TTS) models bring significant advantages over conventional or pre-scripted NPCs for facilitating more natural conversational user interfaces (CUIs) in XR. This paper provides the community with an open-source, customizable, extendable, and privacy-aware Unity package, CUIfy, that facilitates speech-based NPC-user interaction with widely used LLMs, STT, and TTS models. Our package also supports multiple LLM-powered NPCs per environment and minimizes latency between different computational models through streaming to achieve usable interactions between users and NPCs. We publish our source code in the following repository: https://gitlab.lrz.de/hctl/cuify