# Multimodal Recommendation Dialog with Subjective Preference: A New Challenge and Benchmark

arXiv: https://arxiv.org/abs/2305.18212 · HTML: https://mlsanity.com/paper/2305.18212/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 28 references that are in the map.
- First posted on arXiv 26 May 2023.

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [ConceptNet 5.5: An Open Multilingual Graph of General Knowledge](https://mlsanity.com/paper/1612.03975/) (2016)
- [OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework](https://mlsanity.com/paper/2202.03052/) (2022)
- [Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset](https://mlsanity.com/paper/1909.05855/) (2019)
- [Towards Deep Conversational Recommendations](https://mlsanity.com/paper/1812.07617/) (2018)
- [Towards Conversational Recommendation over Multi-Type Dialogs](https://mlsanity.com/paper/2005.03954/) (2020)
- [Towards Topic-Guided Conversational Recommender System](https://mlsanity.com/paper/2010.04125/) (2020)
- [INSPIRED: Toward Sociable Recommendation Dialog Systems](https://mlsanity.com/paper/2009.14306/) (2020)
- [Building a Conversational Agent Overnight with Dialogue Self-Play](https://mlsanity.com/paper/1801.04871/) (2018)
- [SIMMC 2.0: A Task-oriented Dialog Dataset for Immersive Multimodal Conversations](https://mlsanity.com/paper/2104.08667/) (2021)
- [Overview of the Ninth Dialog System Technology Challenge: DSTC9](https://mlsanity.com/paper/2011.06486/) (2020)
- [Recommendation as a Communication Game: Self-Supervised Bot-Play for Goal-oriented Dialogue](https://mlsanity.com/paper/1909.03922/) (2019)
- [Situated and Interactive Multimodal Conversations](https://mlsanity.com/paper/2006.01460/) (2020)
- [SPRING: Situated Conversation Agent Pretrained with Multimodal Questions from Incremental Layout Graph](https://mlsanity.com/paper/2301.01949/) (2023)
- [A Response Retrieval Approach for Dialogue Using a Multi-Attentive Transformer](https://mlsanity.com/paper/2012.08148/) (2020)
- [Multi-Task Learning for Situated Multi-Domain End-to-End Dialogue Systems](https://mlsanity.com/paper/2110.05221/) (2021)

## Abstract (arXiv)

Existing multimodal task-oriented dialog data fails to demonstrate the diverse expressions of user subjective preferences and recommendation acts in the real-life shopping scenario. This paper introduces a new dataset SURE (Multimodal Recommendation Dialog with SUbjective PREference), which contains 12K shopping dialogs in complex store scenes. The data is built in two phases with human annotations to ensure quality and diversity. SURE is well-annotated with subjective preferences and recommendation acts proposed by sales experts. A comprehensive analysis is given to reveal the distinguishing features of SURE. Three benchmark tasks are then proposed on the data to evaluate the capability of multimodal recommendation agents. Based on the SURE, we propose a baseline model, powered by a state-of-the-art multimodal model, for these tasks.