# PRISM-XR: Empowering Privacy-Aware XR Collaboration with Multimodal Large Language Models

arXiv: https://arxiv.org/abs/2602.10154 · HTML: https://mlsanity.com/paper/2602.10154/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 20 of its 67 references that are in the map.
- First posted on arXiv 9 Feb 2026.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM jailbreaks and adversarial robustness](https://mlsanity.com/mllm/mllm-jailbreaks-and-adversarial-robustness/)

## Builds on (most cited first, 20 of 20)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [Voyager: An Open-Ended Embodied Agent with Large Language Models](https://mlsanity.com/paper/2305.16291/) (2023)
- [Large Language Models Can Be Easily Distracted by Irrelevant Context](https://mlsanity.com/paper/2302.00093/) (2023)
- [Agent Laboratory: Using LLM Agents as Research Assistants](https://mlsanity.com/paper/2501.04227/) (2025)
- [A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly](https://mlsanity.com/paper/2312.02003/) (2023)
- [Language to Rewards for Robotic Skill Synthesis](https://mlsanity.com/paper/2306.08647/) (2023)
- [Emergent autonomous scientific research capabilities of large language models](https://mlsanity.com/paper/2304.05332/) (2023)
- [MLR-Copilot: Autonomous Machine Learning Research based on Large Language Models Agents](https://mlsanity.com/paper/2408.14033/) (2024)
- [Prompt a Robot to Walk with Large Language Models](https://mlsanity.com/paper/2309.09969/) (2023)
- [LLMR: Real-time Prompting of Interactive Worlds using Large Language Models](https://mlsanity.com/paper/2309.12276/) (2023)
- [DreamCraft: Text-Guided Generation of Functional 3D Environments in Minecraft](https://mlsanity.com/paper/2404.15538/) (2024)
- [ViDDAR: Vision Language Model-Based Task-Detrimental Content Detection for Augmented Reality](https://mlsanity.com/paper/2501.12553/) (2025)
- [LLMER: Crafting Interactive Extended Reality Worlds with JSON Data Generated by Large Language Models](https://mlsanity.com/paper/2502.02441/) (2025)
- [Privacy Preservation in Artificial Intelligence and Extended Reality (AI-XR) Metaverses: A Survey](https://mlsanity.com/paper/2310.10665/) (2023)
- [Social Conjuring: Multi-User Runtime Collaboration with AI in Building Virtual 3D Worlds](https://mlsanity.com/paper/2410.00274/) (2024)
- [Privacy in Immersive Extended Reality: Exploring User Perceptions, Concerns, and Coping Strategies](https://mlsanity.com/paper/2503.21010/) (2025)
- [Range-based Coordinate Alignment for Cooperative Mobile Sensor Network Localization](https://mlsanity.com/paper/1812.04201/) (2018)
- [When Generative AI Meets Extended Reality: Enabling Scalable and Natural Interactions](https://mlsanity.com/paper/2601.15308/) (2026)

## Built on by (most cited first, 1 of 1)

- [CAPED: Context-Aware Privacy Exposure Defense for Mobile GUI Agents](https://mlsanity.com/paper/2606.12666/) (2026)

## Abstract (arXiv)

Multimodal Large Language Models (MLLMs) enhance collaboration in Extended Reality (XR) environments by enabling flexible object and animation creation through the combination of natural language and visual inputs. However, visual data captured by XR headsets includes real-world backgrounds that may contain irrelevant or sensitive user information, such as credit cards left on the table or facial identities of other users. Uploading those frames to cloud-based MLLMs poses serious privacy risks, particularly when such data is processed without explicit user consent. Additionally, existing colocation and synchronization mechanisms in commercial XR APIs rely on time-consuming, privacy-invasive environment scanning and struggle to adapt to the highly dynamic nature of MLLM-integrated XR environments. In this paper, we propose PRISM-XR, a novel framework that facilitates multi-user collaboration in XR by providing privacy-aware MLLM integration. PRISM-XR employs intelligent frame preprocessing on the edge server to filter sensitive data and remove irrelevant context before communicating with cloud generative AI models. Additionally, we introduce a lightweight registration process and a fully customizable content-sharing mechanism to enable efficient, accurate, and privacy-preserving content synchronization among users. Our numerical evaluation results indicate that the proposed platform achieves nearly 90% accuracy in fulfilling user requests and less than 0.27 seconds registration time while maintaining spatial inconsistencies of less than 3.5 cm. Furthermore, we conducted an IRB-approved user study with 28 participants, demonstrating that our system could automatically filter highly sensitive objects in over 90% of scenarios while maintaining strong overall usability.