# Robust and Interpretable Grounding of Spatial References with Relation Networks

arXiv: https://arxiv.org/abs/2005.00696 · HTML: https://mlsanity.com/paper/2005.00696/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 May 2020.

## Built on by (most cited first, 8 of 8)

- [StepGame: A New Benchmark for Robust Multi-Hop Spatial Reasoning in Texts](https://mlsanity.com/paper/2204.08292/) (2022)
- [Grounding 'Grounding' in NLP](https://mlsanity.com/paper/2106.02192/) (2021)
- [Learning to Execute Actions or Ask Clarification Questions](https://mlsanity.com/paper/2204.08373/) (2022)
- [Leveraging Language for Accelerated Learning of Tool Manipulation](https://mlsanity.com/paper/2206.13074/) (2022)
- [A Survey on Interpretable Cross-modal Reasoning](https://mlsanity.com/paper/2309.01955/) (2023)
- [A Hybrid Model of Classification and Generation for Spatial Relation Extraction](https://mlsanity.com/paper/2208.06961/) (2022)
- [Solving Dialogue Grounding Embodied Task in a Simulated Environment using Further Masked Language Modeling](https://mlsanity.com/paper/2306.12387/) (2023)
- [Improving Agent Interactions in Virtual Environments with Language Models](https://mlsanity.com/paper/2402.05440/) (2024)

## Abstract (arXiv)

Learning representations of spatial references in natural language is a key challenge in tasks like autonomous navigation and robotic manipulation. Recent work has investigated various neural architectures for learning multi-modal representations for spatial concepts. However, the lack of explicit reasoning over entities makes such approaches vulnerable to noise in input text or state observations. In this paper, we develop effective models for understanding spatial references in text that are robust and interpretable, without sacrificing performance. We design a text-conditioned textitrelation network whose parameters are dynamically computed with a cross-modal attention module to capture fine-grained spatial relations between entities. This design choice provides interpretability of learned intermediate outputs. Experiments across three tasks demonstrate that our model achieves superior performance, with a 17% improvement in predicting goal locations and a 15% improvement in robustness compared to state-of-the-art systems.