# Common Ground Tracking in Multimodal Dialogue

arXiv: https://arxiv.org/abs/2403.17284 · HTML: https://mlsanity.com/paper/2403.17284/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 79 references that are in the map.
- First posted on arXiv 26 Mar 2024.

## Builds on (most cited first, 22 of 22)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling](https://mlsanity.com/paper/1810.00278/) (2018)
- [Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding](https://mlsanity.com/paper/1604.01753/) (2016)
- [AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actions](https://mlsanity.com/paper/1705.08421/) (2017)
- [Transferable Multi-Domain State Generator for Task-Oriented Dialogue Systems](https://mlsanity.com/paper/1905.08743/) (2019)
- [Neural Belief Tracker: Data-Driven Dialogue State Tracking](https://mlsanity.com/paper/1606.03777/) (2016)
- [TripPy: A Triple Copy Strategy for Value Independent Neural Dialog State Tracking](https://mlsanity.com/paper/2005.02877/) (2020)
- [The AVA-Kinetics Localized Human Actions Video Dataset](https://mlsanity.com/paper/2005.00214/) (2020)
- [Recent Advances and Challenges in Task-oriented Dialog System](https://mlsanity.com/paper/2003.07490/) (2020)
- [Find or Classify? Dual Strategy for Slot-Value Predictions on Multi-Domain Dialog State Tracking](https://mlsanity.com/paper/1910.03544/) (2019)
- [Dialog State Tracking: A Neural Reading Comprehension Approach](https://mlsanity.com/paper/1908.01946/) (2019)
- [Visual Semantic Role Labeling for Video Understanding](https://mlsanity.com/paper/2104.00990/) (2021)
- [Conversational Semantic Parsing for Dialog State Tracking](https://mlsanity.com/paper/2010.12770/) (2020)
- [Knowledge-Aware Graph-Enhanced GPT-2 for Dialogue State Tracking](https://mlsanity.com/paper/2104.04466/) (2021)
- [HyST: A Hybrid Approach for Flexible and Accurate Dialogue State Tracking](https://mlsanity.com/paper/1907.00883/) (2019)
- [Emergence of hierarchical reference systems in multi-agent communication](https://mlsanity.com/paper/2203.13176/) (2022)
- [VoxML: A Visualization Modeling Language](https://mlsanity.com/paper/1610.01508/) (2016)
- [Evidence and plausibility in neighborhood structures](https://mlsanity.com/paper/1307.1277/) (2013)
- [A Formal Analysis of Multimodal Referring Strategies Under Common Ground](https://mlsanity.com/paper/2003.07385/) (2020)
- [From Rewriting to Remembering: Common Ground for Conversational QA Models](https://mlsanity.com/paper/2204.03930/) (2022)
- [How Good is Automatic Segmentation as a Multimodal Discourse Annotation Aid?](https://mlsanity.com/paper/2305.17350/) (2023)

## Built on by (most cited first, 13 of 13)

- [Frictional Agent Alignment Framework: Slow Down and Don't Break Things](https://mlsanity.com/paper/2505.19428/) (2025)
- [Collaborate, Deliberate, Evaluate: How LLM Alignment Affects Coordinated Multi-Agent Outcomes](https://mlsanity.com/paper/2509.05882/) (2025)
- [Any Other Thoughts, Hedgehog? Linking Deliberation Chains in Collaborative Dialogues](https://mlsanity.com/paper/2410.19301/) (2024)
- [Understanding Common Ground Misalignment in Goal-Oriented Dialog: A Case-Study with Ubuntu Chat Logs](https://mlsanity.com/paper/2503.12370/) (2025)
- [Dynamic Epistemic Friction in Dialogue](https://mlsanity.com/paper/2506.10934/) (2025)
- [Learning "Partner-Aware" Collaborators in Multi-Party Collaboration](https://mlsanity.com/paper/2510.22462/) (2025)
- [Grounded Misunderstandings in Asymmetric Dialogue: A Perspectivist Annotation Scheme for MapTask](https://mlsanity.com/paper/2511.03718/) (2025)
- [Frame of Reference: Addressing the Challenges of Common Ground Representation in Situational Dialogs](https://mlsanity.com/paper/2601.09365/) (2026)
- [Frictive Policy Optimization for LLMs: Epistemic Intervention, Risk-Sensitive Control, and Reflective Alignment](https://mlsanity.com/paper/2604.25136/) (2026)
- [Computer Vision for Objects used in Group Work: Challenges and Opportunities](https://mlsanity.com/paper/2507.00224/) (2025)
- [From Propositional to Perceptual Asymmetry: Extending Frictive Policy Optimization to Asymmetric Partial Information Dialogue](https://mlsanity.com/paper/2606.30973/) (2026)
- [Flout at Your Own Risk: LLMs Struggle with Pragmatic Cooperativity Under Epistemic Asymmetry](https://mlsanity.com/paper/2607.11053/) (2026)
- [Illusion of Alignment: Detecting Hidden Disagreement in Collaborative Dialogue](https://mlsanity.com/paper/2608.08210/) (2026)

## Abstract (arXiv)

Within Dialogue Modeling research in AI and NLP, considerable attention has been spent on ``dialogue state tracking'' (DST), which is the ability to update the representations of the speaker's needs at each turn in the dialogue by taking into account the past dialogue moves and history. Less studied but just as important to dialogue modeling, however, is ``common ground tracking'' (CGT), which identifies the shared belief space held by all of the participants in a task-oriented dialogue: the task-relevant propositions all participants accept as true. In this paper we present a method for automatically identifying the current set of shared beliefs and ``questions under discussion'' (QUDs) of a group with a shared goal. We annotate a dataset of multimodal interactions in a shared physical space with speech transcriptions, prosodic features, gestures, actions, and facets of collaboration, and operationalize these features for use in a deep neural model to predict moves toward construction of common ground. Model outputs cascade into a set of formal closure rules derived from situated evidence and belief axioms and update operations. We empirically assess the contribution of each feature type toward successful construction of common ground relative to ground truth, establishing a benchmark in this novel, challenging task.