# Retargetable AR: Context-aware Augmented Reality in Indoor Scenes based on 3D Scene Graph

arXiv: https://arxiv.org/abs/2008.07817 · HTML: https://mlsanity.com/paper/2008.07817/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Aug 2020.

## Built on by (most cited first, 19 of 19)

- [SGFormer: Semantic Graph Transformer for Point Cloud-based 3D Scene Graph Generation](https://mlsanity.com/paper/2303.11048/) (2023)
- [Indoor and Outdoor 3D Scene Graph Generation via Language-Enabled Spatial Ontologies](https://mlsanity.com/paper/2312.11713/) (2023)
- [3D VSG: Long-term Semantic Scene Change Prediction through 3D Variable Scene Graphs](https://mlsanity.com/paper/2209.07896/) (2022)
- [Universal Scene Graph Generation](https://mlsanity.com/paper/2503.15005/) (2025)
- [SG-PGM: Partial Graph Matching Network with Semantic Geometric Fusion for 3D Scene Graph Alignment and Its Downstream Tasks](https://mlsanity.com/paper/2403.19474/) (2024)
- [Video Perception Models for 3D Scene Synthesis](https://mlsanity.com/paper/2506.20601/) (2025)
- [FlowScene: Style-Consistent Indoor Scene Generation with Multimodal Graph Rectified Flow](https://mlsanity.com/paper/2603.19598/) (2026)
- [Multimodal 3D Fusion and In-Situ Learning for Spatially Aware AI](https://mlsanity.com/paper/2410.04652/) (2024)
- [Event-Driven Storytelling with Multiple Lifelike Humans in a 3D Scene](https://mlsanity.com/paper/2507.19232/) (2025)
- [Object-Centric Representation Learning for Enhanced 3D Semantic Scene Graph Prediction](https://mlsanity.com/paper/2510.04714/) (2025)
- [CleAR: Robust Context-Guided Generative Lighting Estimation for Mobile Augmented Reality](https://mlsanity.com/paper/2411.02179/) (2024)
- [SceneLinker: Compositional 3D Scene Generation via Semantic Scene Graph from RGB Sequences](https://mlsanity.com/paper/2602.02974/) (2026)
- [Generative Augmented Reality: Paradigms, Technologies, and Future Applications](https://mlsanity.com/paper/2511.16783/) (2025)
- [DeWorldSG: Depth-Aware 3D Semantic Scene Graph Generation via World-Model Priors](https://mlsanity.com/paper/2607.00889/) (2026)
- [Optimization of Private Semantic Communication Performance: An Uncooperative Covert Communication Method](https://mlsanity.com/paper/2508.07586/) (2025)
- [Extend Your Horizon: A Device-Agnostic Surgical Tool Tracking Framework with Multi-View Optimization for Augmented Reality](https://mlsanity.com/paper/2603.07981/) (2026)
- [SGFormer++: Semantic Graph Transformer for Incremental 3D Scene Graph Generation](https://mlsanity.com/paper/2606.15328/) (2026)
- [From Scene-Centric to Observer-Centric: Modeling Observer-Aware Relations for 3D Scene Graph Generation](https://mlsanity.com/paper/2606.27412/) (2026)
- [SnapPhysics: A Physics-Aware Scene Graph from a Single View for Interactive Mixed Reality Scenes](https://mlsanity.com/paper/2609.19815/) (2026)

## Abstract (arXiv)

In this paper, we present Retargetable AR, a novel AR framework that yields an AR experience that is aware of scene contexts set in various real environments, achieving natural interaction between the virtual and real worlds. To this end, we characterize scene contexts with relationships among objects in 3D space, not with coordinates transformations. A context assumed by an AR content and a context formed by a real environment where users experience AR are represented as abstract graph representations, i.e. scene graphs. From RGB-D streams, our framework generates a volumetric map in which geometric and semantic information of a scene are integrated. Moreover, using the semantic map, we abstract scene objects as oriented bounding boxes and estimate their orientations. With such a scene representation, our framework constructs, in an online fashion, a 3D scene graph characterizing the context of a real environment for AR. The correspondence between the constructed graph and an AR scene graph denoting the context of AR content provides a semantically registered content arrangement, which facilitates natural interaction between the virtual and real worlds. We performed extensive evaluations on our prototype system through quantitative evaluation of the performance of the oriented bounding box estimation, subjective evaluation of the AR content arrangement based on constructed 3D scene graphs, and an online AR demonstration. The results of these evaluations showed the effectiveness of our framework, demonstrating that it can provide a context-aware AR experience in a variety of real scenes.