# TRACE: Real-Time Multimodal Common Ground Tracking in Situated Collaborative Dialogues

arXiv: https://arxiv.org/abs/2503.09511 · HTML: https://mlsanity.com/paper/2503.09511/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 58 references that are in the map.
- First posted on arXiv 12 Mar 2025.

## Builds on (most cited first, 14 of 14)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling](https://mlsanity.com/paper/1810.00278/) (2018)
- [Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding](https://mlsanity.com/paper/1604.01753/) (2016)
- [AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actions](https://mlsanity.com/paper/1705.08421/) (2017)
- [Large Language Models Fail on Trivial Alterations to Theory-of-Mind Tasks](https://mlsanity.com/paper/2302.08399/) (2023)
- [Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMs](https://mlsanity.com/paper/2210.13312/) (2022)
- [The AVA-Kinetics Localized Human Actions Video Dataset](https://mlsanity.com/paper/2005.00214/) (2020)
- [DeliData: A dataset for deliberation in multi-party problem solving](https://mlsanity.com/paper/2108.05271/) (2021)
- [Multimodal Grounding for Language Processing](https://mlsanity.com/paper/1806.06371/) (2018)
- [Any Other Thoughts, Hedgehog? Linking Deliberation Chains in Collaborative Dialogues](https://mlsanity.com/paper/2410.19301/) (2024)
- [A Formal Analysis of Multimodal Referring Strategies Under Common Ground](https://mlsanity.com/paper/2003.07385/) (2020)
- [Speech Is Not Enough: Interpreting Nonverbal Indicators of Common Knowledge and Engagement](https://mlsanity.com/paper/2412.05797/) (2024)

## Built on by (most cited first, 9 of 9)

- [Frictional Agent Alignment Framework: Slow Down and Don't Break Things](https://mlsanity.com/paper/2505.19428/) (2025)
- [Collaborate, Deliberate, Evaluate: How LLM Alignment Affects Coordinated Multi-Agent Outcomes](https://mlsanity.com/paper/2509.05882/) (2025)
- [Dynamic Epistemic Friction in Dialogue](https://mlsanity.com/paper/2506.10934/) (2025)
- [Learning "Partner-Aware" Collaborators in Multi-Party Collaboration](https://mlsanity.com/paper/2510.22462/) (2025)
- [Distributed Partial Information Puzzles: Examining Common Ground Construction Under Epistemic Asymmetry](https://mlsanity.com/paper/2603.05450/) (2026)
- [Computer Vision for Objects used in Group Work: Challenges and Opportunities](https://mlsanity.com/paper/2507.00224/) (2025)
- [Unified Agent: Managing Interactions across Devices](https://mlsanity.com/paper/2608.05729/) (2026)
- [Illusion of Alignment: Detecting Hidden Disagreement in Collaborative Dialogue](https://mlsanity.com/paper/2608.08210/) (2026)
- [Re2A: Situated Conversational Recommendation via Rubric-based Preference Reasoning and Alignment](https://mlsanity.com/paper/2609.18249/) (2026)

## Abstract (arXiv)

We present TRACE, a novel system for live *common ground* tracking in situated collaborative tasks. With a focus on fast, real-time performance, TRACE tracks the speech, actions, gestures, and visual attention of participants, uses these multimodal inputs to determine the set of task-relevant propositions that have been raised as the dialogue progresses, and tracks the group's epistemic position and beliefs toward them as the task unfolds. Amid increased interest in AI systems that can mediate collaborations, TRACE represents an important step forward for agents that can engage with multiparty, multimodal discourse.