# VRCopilot: Authoring 3D Layouts with Generative AI Models in VR

arXiv: https://arxiv.org/abs/2408.09382 · HTML: https://mlsanity.com/paper/2408.09382/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 65 references that are in the map.
- First posted on arXiv 18 Aug 2024.

## Builds on (most cited first, 19 of 19)

- [Zero-1-to-3: Zero-shot One Image to 3D Object](https://mlsanity.com/paper/2303.11328/) (2023)
- [A Survey Of Methods For Explaining Black Box Models](https://mlsanity.com/paper/1802.01933/) (2018)
- [Objaverse-XL: A Universe of 10M+ 3D Objects](https://mlsanity.com/paper/2307.05663/) (2023)
- [Generative Adversarial Text to Image Synthesis](https://mlsanity.com/paper/1605.05396/) (2016)
- [LION: Latent Point Diffusion Models for 3D Shape Generation](https://mlsanity.com/paper/2210.06978/) (2022)
- [Semantic Scene Completion from a Single Depth Image](https://mlsanity.com/paper/1611.08974/) (2016)
- [GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Images](https://mlsanity.com/paper/2209.11163/) (2022)
- [3D-FRONT: 3D Furnished Rooms with layOuts and semaNTics](https://mlsanity.com/paper/2011.09127/) (2020)
- [ATISS: Autoregressive Transformers for Indoor Scene Synthesis](https://mlsanity.com/paper/2110.03675/) (2021)
- [DiffuScene: Denoising Diffusion Models for Generative Indoor Scene Synthesis](https://mlsanity.com/paper/2303.14207/) (2023)
- [SceneFormer: Indoor Scene Generation with Transformers](https://mlsanity.com/paper/2012.09793/) (2020)
- [Fast and Flexible Indoor Scene Synthesis via Deep Convolutional Generative Models](https://mlsanity.com/paper/1811.12463/) (2018)
- [LEGO-Net: Learning Regular Rearrangements of Objects in Rooms](https://mlsanity.com/paper/2301.09629/) (2023)
- [PromptPaint: Steering Text-to-Image Generation Through Paint Medium-like Interactions](https://mlsanity.com/paper/2308.05184/) (2023)
- [Friend, Collaborator, Student, Manager: How Design of an AI-Driven Game Level Editor Affects Creators](https://mlsanity.com/paper/1901.06417/) (2019)
- [Designerly Understanding: Information Needs for Model Transparency to Support Design Ideation for AI-Powered User Experience](https://mlsanity.com/paper/2302.10395/) (2023)
- [How the Design of YouTube Influences User Sense of Agency](https://mlsanity.com/paper/2101.11778/) (2021)
- [Nine Potential Pitfalls when Designing Human-AI Co-Creative Systems](https://mlsanity.com/paper/2104.00358/) (2021)
- [Scout: Rapid Exploration of Interface Layout Alternatives through High-Level Design Constraints](https://mlsanity.com/paper/2001.05424/) (2020)

## Built on by (most cited first, 10 of 10)

- [GPT for Games: An Updated Scoping Review (2020-2024)](https://mlsanity.com/paper/2411.00308/) (2024)
- [Scaffolding Recursive Divergence and Convergence in Story Ideation](https://mlsanity.com/paper/2507.03307/) (2025)
- [Dreamcrafter: Immersive Editing of 3D Radiance Fields Through Flexible, Generative Inputs and Outputs](https://mlsanity.com/paper/2512.20129/) (2025)
- [MS2Mesh-XR: Multi-modal Sketch-to-Mesh Generation in XR Environments](https://mlsanity.com/paper/2412.09008/) (2024)
- [AccioScene: Compositional 3D Scene Generation via Graph Diffusion and Interaction-driven Critics](https://mlsanity.com/paper/2502.06819/) (2025)
- [Auditing and Controlling AI Agent Actions in Spreadsheets](https://mlsanity.com/paper/2604.20070/) (2026)
- [Empowering Children to Create AI-Enabled Augmented Reality Experiences](https://mlsanity.com/paper/2508.08467/) (2025)
- [MAIC-UI: Making Interactive Courseware with Generative UI](https://mlsanity.com/paper/2604.25806/) (2026)
- [Compos3D: Interactive Part-Based Composition for Creative Control in Generative 3D Models](https://mlsanity.com/paper/2607.12193/) (2026)
- [Generative AI and Extended Reality in Collaborative Architectural Design Education: An Exploratory Studio Study](https://mlsanity.com/paper/2609.13494/) (2026)

## Abstract (arXiv)

Immersive authoring provides an intuitive medium for users to create 3D scenes via direct manipulation in Virtual Reality (VR). Recent advances in generative AI have enabled the automatic creation of realistic 3D layouts. However, it is unclear how capabilities of generative AI can be used in immersive authoring to support fluid interactions, user agency, and creativity. We introduce VRCopilot, a mixed-initiative system that integrates pre-trained generative AI models into immersive authoring to facilitate human-AI co-creation in VR. VRCopilot presents multimodal interactions to support rapid prototyping and iterations with AI, and intermediate representations such as wireframes to augment user controllability over the created content. Through a series of user studies, we evaluated the potential and challenges in manual, scaffolded, and automatic creation in immersive authoring. We found that scaffolded creation using wireframes enhanced the user agency compared to automatic creation. We also found that manual creation via multimodal specification offers the highest sense of creativity and agency.