# Embedding Large Language Models into Extended Reality: Opportunities and Challenges for Inclusion, Engagement, and Privacy

arXiv: https://arxiv.org/abs/2402.03907 · HTML: https://mlsanity.com/paper/2402.03907/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 78 references that are in the map.
- First posted on arXiv 6 Feb 2024.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [Scaling Speech Technology to 1,000+ Languages](https://mlsanity.com/paper/2305.13516/) (2023)
- [ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledge](https://mlsanity.com/paper/2303.14070/) (2023)
- [InferFix: End-to-End Program Repair with LLMs](https://mlsanity.com/paper/2303.07263/) (2023)
- ["It's a Fair Game", or Is It? Examining How Users Navigate Disclosure Risks and Benefits When Using LLM-Based Conversational Agents](https://mlsanity.com/paper/2309.11653/) (2023)
- [Privacy Issues in Large Language Models: A Survey](https://mlsanity.com/paper/2312.06717/) (2023)
- [Identifying and Mitigating Privacy Risks Stemming from Language Models: A Survey](https://mlsanity.com/paper/2310.01424/) (2023)
- [A privacy-preserving approach to streaming eye-tracking data](https://mlsanity.com/paper/2102.01770/) (2021)
- [Eye-tracked Virtual Reality: A Comprehensive Survey on Methods and Privacy Challenges](https://mlsanity.com/paper/2305.14080/) (2023)
- [Differential Privacy for Eye Tracking with Temporal Correlations](https://mlsanity.com/paper/2002.08972/) (2020)
- [Digital Transformations of Classrooms in Virtual Reality](https://mlsanity.com/paper/2101.09576/) (2021)
- [Human-Centered Privacy Research in the Age of Large Language Models](https://mlsanity.com/paper/2402.01994/) (2024)
- [Going Incognito in the Metaverse: Achieving Theoretically Optimal Privacy-Usability Tradeoffs in VR](https://mlsanity.com/paper/2208.05604/) (2022)
- [DiVRsify: Break the Cycle and Develop VR for Everyone](https://mlsanity.com/paper/2110.00497/) (2021)
- [Towards Everyday Virtual Reality through Eye Tracking](https://mlsanity.com/paper/2203.15703/) (2022)
- [Deep Motion Masking for Secure, Usable, and Scalable Real-Time Anonymization of Virtual Reality Motion Data](https://mlsanity.com/paper/2311.05090/) (2023)

## Built on by (most cited first, 14 of 14)

- [How Far Are We From AGI: Are LLMs All We Need?](https://mlsanity.com/paper/2405.10313/) (2024)
- [Space3D-Bench: Spatial 3D Question Answering Benchmark](https://mlsanity.com/paper/2408.16662/) (2024)
- [CUIfy the XR: An Open-Source Package to Embed LLM-powered Conversational Agents in XR](https://mlsanity.com/paper/2411.04671/) (2024)
- [Evaluating Usability and Engagement of Large Language Models in Virtual Reality for Traditional Scottish Curling](https://mlsanity.com/paper/2408.09285/) (2024)
- [Analyzing Multimodal Interaction Strategies for LLM-Assisted Manipulation of 3D Scenes](https://mlsanity.com/paper/2410.22177/) (2024)
- [Exploring Collaborative GenAI Agents in Synchronous Group Settings: Eliciting Team Perceptions and Design Considerations for the Future of Work](https://mlsanity.com/paper/2504.14779/) (2025)
- [Perspective-Aware AI in Extended Reality](https://mlsanity.com/paper/2507.11479/) (2025)
- [Exploring Context-aware and LLM-driven Locomotion for Immersive Virtual Reality](https://mlsanity.com/paper/2504.17331/) (2025)
- [AIvaluateXR: An Evaluation Framework for on-Device AI in XR with Benchmarking Results](https://mlsanity.com/paper/2502.15761/) (2025)
- [Examining the Role of LLM-Driven Interactions on Attention and Cognitive Engagement in Virtual Classrooms](https://mlsanity.com/paper/2505.07377/) (2025)
- [How LLMs are Shaping the Future of Virtual Reality](https://mlsanity.com/paper/2508.00737/) (2025)
- [ORBIT: Guided Agentic Orchestration for Autonomous C-to-Rust Transpilation](https://mlsanity.com/paper/2604.12048/) (2026)
- [Exploring User Acceptance and Concerns toward LLM-powered Conversational Agents in Immersive Extended Reality](https://mlsanity.com/paper/2512.15343/) (2025)
- [AI-Driven Modular Services for Accessible Multilingual Education in Immersive Extended Reality Settings: Integrating Speech Processing, Translation, and Sign Language Rendering](https://mlsanity.com/paper/2604.05591/) (2026)

## Abstract (arXiv)

Advances in artificial intelligence and human-computer interaction will likely lead to extended reality (XR) becoming pervasive. While XR can provide users with interactive, engaging, and immersive experiences, non-player characters are often utilized in pre-scripted and conventional ways. This paper argues for using large language models (LLMs) in XR by embedding them in avatars or as narratives to facilitate inclusion through prompt engineering and fine-tuning the LLMs. We argue that this inclusion will promote diversity for XR use. Furthermore, the versatile conversational capabilities of LLMs will likely increase engagement in XR, helping XR become ubiquitous. Lastly, we speculate that combining the information provided to LLM-powered spaces by users and the biometric data obtained might lead to novel privacy invasions. While exploring potential privacy breaches, examining user privacy concerns and preferences is also essential. Therefore, despite challenges, LLM-powered XR is a promising area with several opportunities.