# Natural Language and Spatial Rules

arXiv: https://arxiv.org/abs/2111.14066 · HTML: https://mlsanity.com/paper/2111.14066/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 2 of its 30 references that are in the map.
- First posted on arXiv 28 Nov 2021.

## Builds on (most cited first, 2 of 2)

- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning](https://mlsanity.com/paper/1612.06890/) (2016)

## Abstract (arXiv)

We develop a system that formally represents spatial semantics concepts within natural language descriptions of spatial arrangements. The system builds on a model of spatial semantics representation according to which words in a sentence are assigned spatial roles and the relations among these roles are represented with spatial relations. We combine our system with the shape grammar formalism that uses shape rules to generate languages (sets) of two-dimensional shapes. Our proposed system consists of pairs of shape rules and verbal rules where the verbal rules describe in English the action of the associated shape rule. We present various types of natural language descriptions of shapes that are successfully parsed by our system and we discuss open questions and challenges we see at the interface of language and perception.