# SceneGen: Generative Contextual Scene Augmentation using Scene Graph Priors

arXiv: https://arxiv.org/abs/2009.12395 · HTML: https://mlsanity.com/paper/2009.12395/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Sep 2020.

## Built on by (most cited first, 11 of 11)

- [ProcTHOR: Large-Scale Embodied AI Using Procedural Generation](https://mlsanity.com/paper/2206.06994/) (2022)
- [ATISS: Autoregressive Transformers for Indoor Scene Synthesis](https://mlsanity.com/paper/2110.03675/) (2021)
- [MIME: Human-Aware 3D Scene Generation](https://mlsanity.com/paper/2212.04360/) (2022)
- [SceneEval: Evaluating Semantic Coherence in Text-Conditioned 3D Indoor Scene Synthesis](https://mlsanity.com/paper/2503.14756/) (2025)
- [Learning 3D Scene Priors with 2D Supervision](https://mlsanity.com/paper/2211.14157/) (2022)
- [HSM: Hierarchical Scene Motifs for Multi-Scale Indoor Scene Generation](https://mlsanity.com/paper/2503.16848/) (2025)
- [CLIP-Layout: Style-Consistent Indoor Scene Synthesis with Semantic Furniture Embedding](https://mlsanity.com/paper/2303.03565/) (2023)
- [Rearrange Indoor Scenes for Human-Robot Co-Activity](https://mlsanity.com/paper/2303.05676/) (2023)
- [Mutual Scene Synthesis for Mixed Reality Telepresence](https://mlsanity.com/paper/2204.00161/) (2022)
- [Contextual Scene Augmentation and Synthesis via GSACNet](https://mlsanity.com/paper/2103.15369/) (2021)
- [Structured Graph Variational Autoencoders for Indoor Furniture layout Generation](https://mlsanity.com/paper/2204.04867/) (2022)

## Abstract (arXiv)

Spatial computing experiences are constrained by the real-world surroundings of the user. In such experiences, augmenting virtual objects to existing scenes require a contextual approach, where geometrical conflicts are avoided, and functional and plausible relationships to other objects are maintained in the target environment. Yet, due to the complexity and diversity of user environments, automatically calculating ideal positions of virtual content that is adaptive to the context of the scene is considered a challenging task. Motivated by this problem, in this paper we introduce SceneGen, a generative contextual augmentation framework that predicts virtual object positions and orientations within existing scenes. SceneGen takes a semantically segmented scene as input, and outputs positional and orientational probability maps for placing virtual content. We formulate a novel spatial Scene Graph representation, which encapsulates explicit topological properties between objects, object groups, and the room. We believe providing explicit and intuitive features plays an important role in informative content creation and user interaction of spatial computing settings, a quality that is not captured in implicit models. We use kernel density estimation (KDE) to build a multivariate conditional knowledge model trained using prior spatial Scene Graphs extracted from real-world 3D scanned data. To further capture orientational properties, we develop a fast pose annotation tool to extend current real-world datasets with orientational labels. Finally, to demonstrate our system in action, we develop an Augmented Reality application, in which objects can be contextually augmented in real-time.