# Dreamcrafter: Immersive Editing of 3D Radiance Fields Through Flexible, Generative Inputs and Outputs

arXiv: https://arxiv.org/abs/2512.20129 · HTML: https://mlsanity.com/paper/2512.20129/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 56 references that are in the map.
- First posted on arXiv 23 Dec 2025.

## Builds on (most cited first, 20 of 20)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [Recurrent World Models Facilitate Policy Evolution](https://mlsanity.com/paper/1809.01999/) (2018)
- [LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creation](https://mlsanity.com/paper/2402.05054/) (2024)
- [Nerfstudio: A Modular Framework for Neural Radiance Field Development](https://mlsanity.com/paper/2302.04264/) (2023)
- [Instruct-NeRF2NeRF: Editing 3D Scenes with Instructions](https://mlsanity.com/paper/2303.12789/) (2023)
- [CAT3D: Create Anything in 3D with Multi-View Diffusion Models](https://mlsanity.com/paper/2405.10314/) (2024)
- [Shap-E: Generating Conditional 3D Implicit Functions](https://mlsanity.com/paper/2305.02463/) (2023)
- [GaussianEditor: Swift and Controllable 3D Editing with Gaussian Splatting](https://mlsanity.com/paper/2311.14521/) (2023)
- [GRM: Large Gaussian Reconstruction Model for Efficient 3D Reconstruction and Generation](https://mlsanity.com/paper/2403.14621/) (2024)
- [VR-GS: A Physical Dynamics-Aware Interactive Gaussian Splatting System in Virtual Reality](https://mlsanity.com/paper/2401.16663/) (2024)
- [LLMR: Real-time Prompting of Interactive Worlds using Large Language Models](https://mlsanity.com/paper/2309.12276/) (2023)
- [Application of 3D Gaussian Splatting for Cinematic Anatomy on Consumer Class Devices](https://mlsanity.com/paper/2404.11285/) (2024)
- [WorldSmith: Iterative and Expressive Prompting for World Building with a Generative AI](https://mlsanity.com/paper/2308.13355/) (2023)
- [Style2Fab: Functionality-Aware Segmentation for Fabricating Personalized 3D Models with Generative AI](https://mlsanity.com/paper/2309.06379/) (2023)
- [VRCopilot: Authoring 3D Layouts with Generative AI Models in VR](https://mlsanity.com/paper/2408.09382/) (2024)
- [How People Prompt to Create Interactive VR Scenes](https://mlsanity.com/paper/2402.10525/) (2024)

## Built on by (most cited first, 2 of 2)

- [SemanticSlider3D: Training-Free Continuous Semantic Editing for 3D Objects](https://mlsanity.com/paper/2608.18560/) (2026)
- [Generative Tutorial: Towards Live Contextualized Visual Instructions for Physical Tasks](https://mlsanity.com/paper/2609.24955/) (2026)

## Abstract (arXiv)

Authoring 3D scenes is a central task for spatial computing applications. Competing visions for lowering existing barriers are (1) focus on immersive, direct manipulation of 3D content or (2) leverage AI techniques that capture real scenes (3D Radiance Fields such as, NeRFs, 3D Gaussian Splatting) and modify them at a higher level of abstraction, at the cost of high latency. We unify the complementary strengths of these approaches and investigate how to integrate generative AI advances into real-time, immersive 3D Radiance Field editing. We introduce Dreamcrafter, a VR-based 3D scene editing system that: (1) provides a modular architecture to integrate generative AI algorithms; (2) combines different levels of control for creating objects, including natural language and direct manipulation; and (3) introduces proxy representations that support interaction during high-latency operations. We contribute empirical findings on control preferences and discuss how generative AI interfaces beyond text input enhance creativity in scene editing and world building.