mlsanityData to 14 Oct 2026

MMIU: Dataset for Visual Intent Understanding in Multimodal Assistants

arXiv 2110.06416 ↗First posted 13 Oct 2021cs.CV cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

In multimodal assistant, where vision is also one of the input modalities, the identification of user intent becomes a challenging task as visual input can influence the outcome. Current digital assistants take spoken input and try to determine the user intent from conversational or device context. So, a dataset, which includes visual input (i.e. images or videos for the corresponding questions targeted for multimodal assistant use cases, is not readily available. The research in visual question answering (VQA) and visual question generation (VQG) is a great step forward. However, they do not capture questions that a visually-abled person would ask multimodal assistants. Moreover, many times questions do not seek information from external knowledge. In this paper, we provide a new dataset, MMIU (MultiModal Intent Understanding), that contains questions and corresponding intents provided by human annotators while looking at images. We, then, use this dataset for intent classification task in multimodal digital assistant. We also experiment with various approaches for combining vision and language features including the use of multimodal transformer for classification of image-question pairs into 14 intents. We provide the benchmark results and discuss the role of visual and text features for the intent classification task on our dataset.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 17 of its 19 references that are in the map. First posted on arXiv 13 Oct 2021.

Builds on

17 of 19 references.

References · 17

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2016Densely Connected Convolutional Networkscited by 6,720
  6. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  7. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  8. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  9. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  10. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  11. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  12. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
5 more
  1. 2019UNITER: UNiversal Image-TExt Representation Learningcited by 228
  2. 2020Metrics for Multi-Class Classification: an Overviewcited by 143
  3. 2016Generating Natural Questions About an Imagecited by 56
  4. 2018Improved Fusion of Visual and Language Representations by Dense Symmetric Co-Attention for Visual Question Answeringcited by 51
  5. 2017Creativity: Generating Diverse Questions using Variational Autoencoderscited by 24

Built on by

Not cited yet.