# Spatial Commonsense Graph for Object Localisation in Partial Scenes arXiv: https://arxiv.org/abs/2203.05380 ยท HTML: https://mlsanity.com/paper/2203.05380/ ## Key facts - Cited by 9 papers in the map (data through 14 Oct 2026). - Builds on 32 of its 43 references that are in the map. - First posted on arXiv 10 Mar 2022. ## Builds on (most cited first, 30 of 32) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017) - [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017) - [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016) - [ConceptNet 5.5: An Open Multilingual Graph of General Knowledge](https://mlsanity.com/paper/1612.03975/) (2016) - [Strategies for Pre-training Graph Neural Networks](https://mlsanity.com/paper/1905.12265/) (2019) - [Adafactor: Adaptive Learning Rates with Sublinear Memory Cost](https://mlsanity.com/paper/1804.04235/) (2018) - [How Attentive are Graph Attention Networks?](https://mlsanity.com/paper/2105.14491/) (2021) - [Heterogeneous Graph Attention Network](https://mlsanity.com/paper/1903.07293/) (2019) - [Open3D: A Modern Library for 3D Data Processing](https://mlsanity.com/paper/1801.09847/) (2018) - [Measuring and Relieving the Over-smoothing Problem for Graph Neural Networks from the Topological View](https://mlsanity.com/paper/1909.03211/) (2019) - [Masked Label Prediction: Unified Message Passing Model for Semi-Supervised Classification](https://mlsanity.com/paper/2009.03509/) (2020) - [Object Goal Navigation using Goal-Oriented Semantic Exploration](https://mlsanity.com/paper/2007.00643/) (2020) - [3D Scene Graph: A Structure for Unified Semantics, 3D Space, and Camera](https://mlsanity.com/paper/1910.02527/) (2019) - [Auto-Encoding Scene Graphs for Image Captioning](https://mlsanity.com/paper/1812.02378/) (2018) - [ObjectNav Revisited: On Evaluation of Embodied Agents Navigating to Objects](https://mlsanity.com/paper/2006.13171/) (2020) - [Learning 3D Semantic Scene Graphs from 3D Indoor Reconstructions](https://mlsanity.com/paper/2004.03967/) (2020) - [SceneGraphFusion: Incremental 3D Scene Graph Prediction from RGB-D Sequences](https://mlsanity.com/paper/2103.14898/) (2021) - [LayoutTransformer: Layout Generation and Completion with Self-attention](https://mlsanity.com/paper/2006.14615/) (2020) - [Explainable and Explicit Visual Reasoning over Scene Graphs](https://mlsanity.com/paper/1812.01855/) (2018) - [Scene Graph Generation with External Knowledge and Image Reconstruction](https://mlsanity.com/paper/1904.00560/) (2019) - [Graph-to-3D: End-to-End Generation and Manipulation of 3D Scenes Using Scene Graphs](https://mlsanity.com/paper/2108.08841/) (2021) - [Unpaired Image Captioning via Scene Graph Alignments](https://mlsanity.com/paper/1903.10658/) (2019) - [SceneGraphNet: Neural Message Passing for 3D Indoor Scene Augmentation](https://mlsanity.com/paper/1907.11308/) (2019) - [End-to-End Optimization of Scene Layout](https://mlsanity.com/paper/2007.11744/) (2020) - [GRAINS: Generative Recursive Autoencoders for INdoor Scenes](https://mlsanity.com/paper/1807.09193/) (2018) - [Visual Graphs from Motion (VGfM): Scene understanding with object geometry reasoning](https://mlsanity.com/paper/1807.05933/) (2018) - [ConceptNet at SemEval-2017 Task 2: Extending Word Embeddings with Multilingual Relational Knowledge](https://mlsanity.com/paper/1704.03560/) (2017) - [Incorporating External Knowledge to Answer Open-Domain Visual Questions with Dynamic Memory Networks](https://mlsanity.com/paper/1712.00733/) (2017) - [KI-BERT: Infusing Knowledge Context for Better Language and Domain Understanding](https://mlsanity.com/paper/2104.08145/) (2021) ## Built on by (most cited first, 9 of 9) - [DiffAssemble: A Unified Graph-Diffusion Model for 2D and 3D Reassembly](https://mlsanity.com/paper/2402.19302/) (2024) - [3D VSG: Long-term Semantic Scene Change Prediction through 3D Variable Scene Graphs](https://mlsanity.com/paper/2209.07896/) (2022) - [SpaGBOL: Spatial-Graph-Based Orientated Localisation](https://mlsanity.com/paper/2409.15514/) (2024) - [Novel class discovery meets foundation models for 3D semantic segmentation](https://mlsanity.com/paper/2312.03782/) (2023) - [Belief Scene Graphs: Expanding Partial Scenes with Objects through Computation of Expectation](https://mlsanity.com/paper/2402.03840/) (2024) - [Leveraging commonsense for object localisation in partial scenes](https://mlsanity.com/paper/2211.00562/) (2022) - [Grounding Scene Graphs on Natural Images via Visio-Lingual Message Passing](https://mlsanity.com/paper/2211.01969/) (2022) - [Maps from Motion (MfM): Generating 2D Semantic Maps from Sparse Multi-view Images](https://mlsanity.com/paper/2411.12620/) (2024) - [Model-based inexact graph matching on top of CNNs for semantic scene understanding](https://mlsanity.com/paper/2301.07468/) (2023) ## Abstract (arXiv) We solve object localisation in partial scenes, a new problem of estimating the unknown position of an object (e.g. where is the bag?) given a partial 3D scan of a scene. The proposed solution is based on a novel scene graph model, the Spatial Commonsense Graph (SCG), where objects are the nodes and edges define pairwise distances between them, enriched by concept nodes and relationships from a commonsense knowledge base. This allows SCG to better generalise its spatial inference over unknown 3D scenes. The SCG is used to estimate the unknown position of the target object in two steps: first, we feed the SCG into a novel Proximity Prediction Network, a graph neural network that uses attention to perform distance prediction between the node representing the target object and the nodes representing the observed objects in the SCG; second, we propose a Localisation Module based on circular intersection to estimate the object position using all the predicted pairwise distances in order to be independent of any reference system. We create a new dataset of partially reconstructed scenes to benchmark our method and baselines for object localisation in partial scenes, where our proposed method achieves the best localisation performance.