# Mutual Scene Synthesis for Mixed Reality Telepresence

arXiv: https://arxiv.org/abs/2204.00161 · HTML: https://mlsanity.com/paper/2204.00161/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 69 references that are in the map.
- First posted on arXiv 1 Apr 2022.

## Builds on (most cited first, 18 of 18)

- [Matterport3D: Learning from RGB-D Data in Indoor Environments](https://mlsanity.com/paper/1709.06158/) (2017)
- [NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections](https://mlsanity.com/paper/2008.02268/) (2020)
- [Advances in Neural Rendering](https://mlsanity.com/paper/2111.05849/) (2021)
- [Volumetric and Multi-View CNNs for Object Classification on 3D Data](https://mlsanity.com/paper/1604.03265/) (2016)
- [ATISS: Autoregressive Transformers for Indoor Scene Synthesis](https://mlsanity.com/paper/2110.03675/) (2021)
- [Deep Appearance Models for Face Rendering](https://mlsanity.com/paper/1808.00362/) (2018)
- [SceneFormer: Indoor Scene Generation with Transformers](https://mlsanity.com/paper/2012.09793/) (2020)
- [Pixel Codec Avatars](https://mlsanity.com/paper/2104.04638/) (2021)
- [Fast and Flexible Indoor Scene Synthesis via Deep Convolutional Generative Models](https://mlsanity.com/paper/1811.12463/) (2018)
- [SceneGraphNet: Neural Message Passing for 3D Indoor Scene Augmentation](https://mlsanity.com/paper/1907.11308/) (2019)
- [Audio- and Gaze-driven Facial Animation of Codec Avatars](https://mlsanity.com/paper/2008.05023/) (2020)
- [FloorNet: A Unified Framework for Floorplan Reconstruction from 3D Scans](https://mlsanity.com/paper/1804.00090/) (2018)
- [SceneGen: Generative Contextual Scene Augmentation using Scene Graph Priors](https://mlsanity.com/paper/2009.12395/) (2020)
- [SLAMCast: Large-Scale, Real-Time 3D Reconstruction and Streaming for Immersive Multi-Client Live Telepresence](https://mlsanity.com/paper/1805.03709/) (2018)
- [Affordance Analysis of Virtual and Augmented Reality Mediated Communication](https://mlsanity.com/paper/1904.04723/) (2019)
- [Optimization and Manipulation of Contextual Mutual Spaces for Multi-User Virtual and Augmented Reality Interaction](https://mlsanity.com/paper/1910.05998/) (2019)
- [GenScan: A Generative Method for Populating Parametric 3D Scan Datasets](https://mlsanity.com/paper/2012.03998/) (2020)
- [Contextual Scene Augmentation and Synthesis via GSACNet](https://mlsanity.com/paper/2103.15369/) (2021)

## Built on by (most cited first, 3 of 3)

- [CLIP-Layout: Style-Consistent Indoor Scene Synthesis with Semantic Furniture Embedding](https://mlsanity.com/paper/2303.03565/) (2023)
- [ROOT: VLM based System for Indoor Scene Understanding and Beyond](https://mlsanity.com/paper/2411.15714/) (2024)
- [Learning 3D Scene Analogies with Neural Contextual Scene Maps](https://mlsanity.com/paper/2503.15897/) (2025)

## Abstract (arXiv)

Remote telepresence via next-generation mixed reality platforms can provide higher levels of immersion for computer-mediated communications, allowing participants to engage in a wide spectrum of activities, previously not possible in 2D screen-based communication methods. However, as mixed reality experiences are limited to the local physical surrounding of each user, finding a common virtual ground where users can freely move and interact with each other is challenging. In this paper, we propose a novel mutual scene synthesis method that takes the participants' spaces as input, and generates a virtual synthetic scene that corresponds to the functional features of all participants' local spaces. Our method combines a mutual function optimization module with a deep-learning conditional scene augmentation process to generate a scene mutually and physically accessible to all participants of a mixed reality telepresence scenario. The synthesized scene can hold mutual walkable, sittable and workable functions, all corresponding to physical objects in the users' real environments. We perform experiments using the MatterPort3D dataset and conduct comparative user studies to evaluate the effectiveness of our system. Our results show that our proposed approach can be a promising research direction for facilitating contextualized telepresence systems for next-generation spatial computing platforms.