# McQueen: a Benchmark for Multimodal Conversational Query Rewrite

arXiv: https://arxiv.org/abs/2210.12775 · HTML: https://mlsanity.com/paper/2210.12775/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 41 references that are in the map.
- First posted on arXiv 23 Oct 2022.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Get To The Point: Summarization with Pointer-Generator Networks](https://mlsanity.com/paper/1704.04368/) (2017)
- [CoQA: A Conversational Question Answering Challenge](https://mlsanity.com/paper/1808.07042/) (2018)
- [Unifying Vision-and-Language Tasks via Text Generation](https://mlsanity.com/paper/2102.02779/) (2021)
- [QuAC : Question Answering in Context](https://mlsanity.com/paper/1808.07036/) (2018)
- [Visual Dialog](https://mlsanity.com/paper/1611.08669/) (2016)
- [AllenNLP: A Deep Semantic Natural Language Processing Platform](https://mlsanity.com/paper/1803.07640/) (2018)
- [Open-Domain Question Answering Goes Conversational via Question Rewriting](https://mlsanity.com/paper/2010.04898/) (2020)
- [Question Rewriting for Conversational Question Answering](https://mlsanity.com/paper/2004.14652/) (2020)
- [Few-Shot Generative Conversational Query Rewriting](https://mlsanity.com/paper/2006.05009/) (2020)
- [Few-Shot Conversational Dense Retrieval](https://mlsanity.com/paper/2105.04166/) (2021)
- [Improving Multi-turn Dialogue Modelling with Utterance ReWriter](https://mlsanity.com/paper/1906.07004/) (2019)
- [Visual Coreference Resolution in Visual Dialog using Neural Module Networks](https://mlsanity.com/paper/1809.01816/) (2018)
- [Query Resolution for Conversational Search with Limited Supervision](https://mlsanity.com/paper/2005.11723/) (2020)
- [GECOR: An End-to-End Generative Ellipsis and Co-reference Resolution Model for Task-Oriented Dialogue](https://mlsanity.com/paper/1909.12086/) (2019)
- [Scaling Multi-Domain Dialogue State Tracking via Query Reformulation](https://mlsanity.com/paper/1903.05164/) (2019)
- [What You See is What You Get: Visual Pronoun Coreference Resolution in Dialogues](https://mlsanity.com/paper/1909.00421/) (2019)
- [Semantic Role Labeling Guided Multi-turn Dialogue ReWriter](https://mlsanity.com/paper/2010.01417/) (2020)
- [Filling Conversation Ellipsis for Better Social Dialog Understanding](https://mlsanity.com/paper/1911.10776/) (2019)
- [VD-PCR: Improving Visual Dialog with Pronoun Coreference Resolution](https://mlsanity.com/paper/2205.14693/) (2022)
- [MLR: A Two-stage Conversational Query Rewriting Model with Multi-task Learning](https://mlsanity.com/paper/2004.05812/) (2020)
- [Modeling Coreference Relations in Visual Dialog](https://mlsanity.com/paper/2203.02986/) (2022)

## Built on by (most cited first, 6 of 6)

- [Enhancing Conversational Search: Large Language Model-Aided Informative Query Rewriting](https://mlsanity.com/paper/2310.09716/) (2023)
- [Asking Multimodal Clarifying Questions in Mixed-Initiative Conversational Search](https://mlsanity.com/paper/2402.07742/) (2024)
- [Query Understanding in LLM-based Conversational Information Seeking](https://mlsanity.com/paper/2504.06356/) (2025)
- [Multi-User MultiWOZ: Task-Oriented Dialogues among Multiple Users](https://mlsanity.com/paper/2310.20479/) (2023)
- [ReCQR: Incorporating conversational query rewriting to improve Multimodal Image Retrieval](https://mlsanity.com/paper/2603.26669/) (2026)
- [CO3: Low-resource Contrastive Co-training for Generative Conversational Query Rewrite](https://mlsanity.com/paper/2403.11873/) (2024)

## Abstract (arXiv)

The task of query rewrite aims to convert an in-context query to its fully-specified version where ellipsis and coreference are completed and referred-back according to the history context. Although much progress has been made, less efforts have been paid to real scenario conversations that involve drawing information from more than one modalities. In this paper, we propose the task of multimodal conversational query rewrite (McQR), which performs query rewrite under the multimodal visual conversation setting. We collect a large-scale dataset named McQueen based on manual annotation, which contains 15k visual conversations and over 80k queries where each one is associated with a fully-specified rewrite version. In addition, for entities appearing in the rewrite, we provide the corresponding image box annotation. We then use the McQueen dataset to benchmark a state-of-the-art method for effectively tackling the McQR task, which is based on a multimodal pre-trained model with pointer generator. Extensive experiments are performed to demonstrate the effectiveness of our model on this taskfootnoteThe dataset and code of this paper are both available in urlhttps://github.com/yfyuan01/MQR