# Human-Robot Dialogue Annotation for Multi-Modal Common Ground

arXiv: https://arxiv.org/abs/2411.12829 · HTML: https://mlsanity.com/paper/2411.12829/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 22 references that are in the map.
- First posted on arXiv 19 Nov 2024.

## Builds on (most cited first, 5 of 5)

- [Generalized Planning in PDDL Domains with Pretrained Large Language Models](https://mlsanity.com/paper/2305.11014/) (2023)
- [A Review of Verbal and Non-Verbal Human-Robot Interactive Communication](https://mlsanity.com/paper/1401.4994/) (2014)
- [DANLI: Deliberative Agent for Following Natural Language Instructions](https://mlsanity.com/paper/2210.12485/) (2022)
- [Design and Implementation of a Maxi-Sized Mobile Robot (Karo) for Rescue Missions](https://mlsanity.com/paper/2008.10396/) (2020)
- [Semantically-informed Hierarchical Event Modeling](https://mlsanity.com/paper/2212.10547/) (2022)

## Built on by (most cited first, 2 of 2)

- [AHA -- Predicting What Matters Next: Online Highlight Detection Without Looking Ahead](https://mlsanity.com/paper/2509.16421/) (2025)
- [FRIDA to the Rescue! Analyzing Synthetic Data Effectiveness in Object-Based Common Sense Reasoning for Disaster Response](https://mlsanity.com/paper/2502.18452/) (2025)

## Abstract (arXiv)

In this paper, we describe the development of symbolic representations annotated on human-robot dialogue data to make dimensions of meaning accessible to autonomous systems participating in collaborative, natural language dialogue, and to enable common ground with human partners. A particular challenge for establishing common ground arises in remote dialogue (occurring in disaster relief or search-and-rescue tasks), where a human and robot are engaged in a joint navigation and exploration task of an unfamiliar environment, but where the robot cannot immediately share high quality visual information due to limited communication constraints. Engaging in a dialogue provides an effective way to communicate, while on-demand or lower-quality visual information can be supplemented for establishing common ground. Within this paradigm, we capture propositional semantics and the illocutionary force of a single utterance within the dialogue through our Dialogue-AMR annotation, an augmentation of Abstract Meaning Representation. We then capture patterns in how different utterances within and across speaker floors relate to one another in our development of a multi-floor Dialogue Structure annotation schema. Finally, we begin to annotate and analyze the ways in which the visual modalities provide contextual information to the dialogue for overcoming disparities in the collaborators' understanding of the environment. We conclude by discussing the use-cases, architectures, and systems we have implemented from our annotations that enable physical robots to autonomously engage with humans in bi-directional dialogue and navigation.