# Incremental Object Grounding Using Scene Graphs

arXiv: https://arxiv.org/abs/2201.01901 · HTML: https://mlsanity.com/paper/2201.01901/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 33 references that are in the map.
- First posted on arXiv 6 Jan 2022.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Scene Graph Generation by Iterative Message Passing](https://mlsanity.com/paper/1701.02426/) (2017)
- [MAttNet: Modular Attention Network for Referring Expression Comprehension](https://mlsanity.com/paper/1801.08186/) (2018)
- [Neural Motifs: Scene Graph Parsing with Global Context](https://mlsanity.com/paper/1711.06640/) (2017)
- [Unbiased Scene Graph Generation from Biased Training](https://mlsanity.com/paper/2002.11949/) (2020)
- [Graph R-CNN for Scene Graph Generation](https://mlsanity.com/paper/1808.00191/) (2018)
- [Auto-Encoding Scene Graphs for Image Captioning](https://mlsanity.com/paper/1812.02378/) (2018)
- [Exploring Visual Relationship for Image Captioning](https://mlsanity.com/paper/1809.07041/) (2018)
- [Graph-Structured Representations for Visual Question Answering](https://mlsanity.com/paper/1609.05600/) (2016)
- [Modeling Relationships in Referential Expressions with Compositional Modular Networks](https://mlsanity.com/paper/1611.09978/) (2016)
- [Semantic Image Manipulation Using Scene Graphs](https://mlsanity.com/paper/2004.03677/) (2020)
- [Interactively Picking Real-World Objects with Unconstrained Spoken Language Instructions](https://mlsanity.com/paper/1710.06280/) (2017)
- [Graph-Structured Referring Expression Reasoning in The Wild](https://mlsanity.com/paper/2004.08814/) (2020)
- [Parallel Attention: A Unified Framework for Visual Object Discovery through Dialogs and Queries](https://mlsanity.com/paper/1711.06370/) (2017)
- [Interactive Visual Grounding of Referring Expressions for Human-Robot Interaction](https://mlsanity.com/paper/1806.03831/) (2018)
- [Scene Graph Reasoning for Visual Question Answering](https://mlsanity.com/paper/2007.01072/) (2020)
- [Relationship-Embedded Representation Learning for Grounding Referring Expressions](https://mlsanity.com/paper/1906.04464/) (2019)
- [Understanding Natural Language Instructions for Fetching Daily Objects Using GAN-Based Multimodal Target-Source Classification](https://mlsanity.com/paper/1906.06830/) (2019)
- [A Comparison of Visualisation Methods for Disambiguating Verbal Requests in Human-Robot Interaction](https://mlsanity.com/paper/1801.08760/) (2018)
- [Open Challenges on Generating Referring Expressions for Human-Robot Interaction](https://mlsanity.com/paper/2104.09193/) (2021)

## Built on by (most cited first, 6 of 6)

- [Reasoning with Scene Graphs for Robot Planning under Partial Observability](https://mlsanity.com/paper/2202.10432/) (2022)
- [Heterogeneous Trajectory Forecasting via Risk and Scene Graph Learning](https://mlsanity.com/paper/2211.00848/) (2022)
- [GVCCI: Lifelong Learning of Visual Grounding for Language-Guided Robotic Manipulation](https://mlsanity.com/paper/2307.05963/) (2023)
- [SGRAM: Improving Scene Graph Parsing via Abstract Meaning Representation](https://mlsanity.com/paper/2210.08675/) (2022)
- [SECURE: Semantics-aware Embodied Conversation under Unawareness for Lifelong Robot Learning](https://mlsanity.com/paper/2409.17755/) (2024)
- [Context-Infused Visual Grounding for Art](https://mlsanity.com/paper/2410.12369/) (2024)

## Abstract (arXiv)

Object grounding tasks aim to locate the target object in an image through verbal communications. Understanding human command is an important process needed for effective human-robot communication. However, this is challenging because human commands can be ambiguous and erroneous. This paper aims to disambiguate the human's referring expressions by allowing the agent to ask relevant questions based on semantic data obtained from scene graphs. We test if our agent can use relations between objects from a scene graph to ask semantically relevant questions that can disambiguate the original user command. In this paper, we present Incremental Grounding using Scene Graphs (IGSG), a disambiguation model that uses semantic data from an image scene graph and linguistic structures from a language scene graph to ground objects based on human command. Compared to the baseline, IGSG shows promising results in complex real-world scenes where there are multiple identical target objects. IGSG can effectively disambiguate ambiguous or wrong referring expressions by asking disambiguating questions back to the user.