# MARRS: Multimodal Reference Resolution System

arXiv: https://arxiv.org/abs/2311.01650 · HTML: https://mlsanity.com/paper/2311.01650/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 12 references that are in the map.
- First posted on arXiv 3 Nov 2023.

## Builds on (most cited first, 9 of 9)

- [MAttNet: Modular Attention Network for Referring Expression Comprehension](https://mlsanity.com/paper/1801.08186/) (2018)
- [End-to-end Neural Coreference Resolution](https://mlsanity.com/paper/1707.07045/) (2017)
- [Incorporating Copying Mechanism in Sequence-to-Sequence Learning](https://mlsanity.com/paper/1603.06393/) (2016)
- [Few-Shot Generative Conversational Query Rewriting](https://mlsanity.com/paper/2006.05009/) (2020)
- [Coreference Resolution through a seq2seq Transition-Based System](https://mlsanity.com/paper/2211.12142/) (2022)
- [GECOR: An End-to-End Generative Ellipsis and Co-reference Resolution Model for Task-Oriented Dialogue](https://mlsanity.com/paper/1909.12086/) (2019)
- [CREAD: Combined Resolution of Ellipses and Anaphora in Dialogues](https://mlsanity.com/paper/2105.09914/) (2021)
- [User-Initiated Repetition-Based Recovery in Multi-Utterance Dialogue Systems](https://mlsanity.com/paper/2108.01208/) (2021)
- [Referring to Screen Texts with Voice Assistants](https://mlsanity.com/paper/2306.07298/) (2023)

## Built on by (most cited first, 1 of 1)

- [Seeing Without Exposing: Adaptive Privacy Control for Open-World, Context-Hungry MLLMs](https://mlsanity.com/paper/2606.07175/) (2026)

## Abstract (arXiv)

Successfully handling context is essential for any dialog understanding task. This context maybe be conversational (relying on previous user queries or system responses), visual (relying on what the user sees, for example, on their screen), or background (based on signals such as a ringing alarm or playing music). In this work, we present an overview of MARRS, or Multimodal Reference Resolution System, an on-device framework within a Natural Language Understanding system, responsible for handling conversational, visual and background context. In particular, we present different machine learning models to enable handing contextual queries; specifically, one to enable reference resolution, and one to handle context via query rewriting. We also describe how these models complement each other to form a unified, coherent, lightweight system that can understand context while preserving user privacy.