# Multi-Turn Multi-Modal Question Clarification for Enhanced Conversational Understanding

arXiv: https://arxiv.org/abs/2502.11442 · HTML: https://mlsanity.com/paper/2502.11442/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 40 references that are in the map.
- First posted on arXiv 17 Feb 2025.

## Builds on (most cited first, 18 of 18)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [VisualBERT: A Simple and Performant Baseline for Vision and Language](https://mlsanity.com/paper/1908.03557/) (2019)
- [Fast Lexically Constrained Decoding with Dynamic Beam Allocation for Neural Machine Translation](https://mlsanity.com/paper/1804.06609/) (2018)
- [Asking Clarifying Questions in Open-Domain Information-Seeking Conversations](https://mlsanity.com/paper/1907.06554/) (2019)
- [Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Information](https://mlsanity.com/paper/1805.04655/) (2018)
- [Learning from Dialogue after Deployment: Feed Yourself, Chatbot!](https://mlsanity.com/paper/1901.05415/) (2019)
- [Building and Evaluating Open-Domain Dialogue Corpora with Clarifying Questions](https://mlsanity.com/paper/2109.05794/) (2021)
- [ConvAI3: Generating Clarifying Questions for Open-Domain Dialogue Systems (ClariQ)](https://mlsanity.com/paper/2009.11352/) (2020)
- [FashionBERT: Text and Image Matching with Adaptive Loss for Cross-modal Retrieval](https://mlsanity.com/paper/2005.09801/) (2020)
- [Learning to Rank in Generative Retrieval](https://mlsanity.com/paper/2306.15222/) (2023)
- [Zero-shot Clarifying Question Generation for Conversational Search](https://mlsanity.com/paper/2301.12660/) (2023)
- [Listwise Generative Retrieval Models via a Sequential Learning Process](https://mlsanity.com/paper/2403.12499/) (2024)
- [Asking Multimodal Clarifying Questions in Mixed-Initiative Conversational Search](https://mlsanity.com/paper/2402.07742/) (2024)
- [AGENT-CQ: Automatic Generation and Evaluation of Clarifying Questions for Conversational Search with LLMs](https://mlsanity.com/paper/2410.19692/) (2024)

## Built on by (most cited first, 3 of 3)

- [ClarifyMT-Bench: Benchmarking and Improving Multi-Turn Clarification for Conversational Large Language Models](https://mlsanity.com/paper/2512.21120/) (2025)
- [AMIGO: Agentic Multi-Image Grounding Oracle Benchmark](https://mlsanity.com/paper/2603.28662/) (2026)
- [Region-R1: Reinforcing Query-Side Region Cropping for Multi-Modal Re-Ranking](https://mlsanity.com/paper/2604.05268/) (2026)

## Abstract (arXiv)

Conversational query clarification enables users to refine their search queries through interactive dialogue, improving search effectiveness. Traditional approaches rely on text-based clarifying questions, which often fail to capture complex user preferences, particularly those involving visual attributes. While recent work has explored single-turn multi-modal clarification with images alongside text, such methods do not fully support the progressive nature of user intent refinement over multiple turns. Motivated by this, we introduce the Multi-turn Multi-modal Clarifying Questions (MMCQ) task, which combines text and visual modalities to refine user queries in a multi-turn conversation. To facilitate this task, we create a large-scale dataset named ClariMM comprising over 13k multi-turn interactions and 33k question-answer pairs containing multi-modal clarifying questions. We propose Mario, a retrieval framework that employs a two-phase ranking strategy: initial retrieval with BM25, followed by a multi-modal generative re-ranking model that integrates textual and visual information from conversational history. Our experiments show that multi-turn multi-modal clarification outperforms uni-modal and single-turn approaches, improving MRR by 12.88%. The gains are most significant in longer interactions, demonstrating the value of progressive refinement for complex queries.