# MIntRec: A New Dataset for Multimodal Intent Recognition

arXiv: https://arxiv.org/abs/2209.04355 · HTML: https://mlsanity.com/paper/2209.04355/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 69 references that are in the map.
- First posted on arXiv 9 Sep 2022.

## Builds on (most cited first, 30 of 39)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [MMDetection: Open MMLab Detection Toolbox and Benchmark](https://mlsanity.com/paper/1906.07155/) (2019)
- [Multimodal Transformer for Unaligned Multimodal Language Sequences](https://mlsanity.com/paper/1906.00295/) (2019)
- [MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations](https://mlsanity.com/paper/1810.02508/) (2018)
- [Tensor Fusion Network for Multimodal Sentiment Analysis](https://mlsanity.com/paper/1707.07250/) (2017)
- [Efficient Intent Detection with Dual Sentence Encoders](https://mlsanity.com/paper/2003.04807/) (2020)
- [An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction](https://mlsanity.com/paper/1909.02027/) (2019)
- [Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfaces](https://mlsanity.com/paper/1805.10190/) (2018)
- [Efficient Low-rank Multimodal Fusion with Modality-Specific Factors](https://mlsanity.com/paper/1806.00064/) (2018)
- [MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2005.03545/) (2020)
- [Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding](https://mlsanity.com/paper/1606.01847/) (2016)
- [BERT for Joint Intent Classification and Slot Filling](https://mlsanity.com/paper/1902.10909/) (2019)
- [MOSI: Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis in Online Opinion Videos](https://mlsanity.com/paper/1606.06259/) (2016)
- [Self-Supervised Learning of Audio-Visual Objects from Video](https://mlsanity.com/paper/2008.04237/) (2020)
- [Towards Multimodal Sarcasm Detection (An _Obviously_ Perfect Paper)](https://mlsanity.com/paper/1906.01815/) (2019)
- [Memory Fusion Network for Multi-view Sequential Learning](https://mlsanity.com/paper/1802.00927/) (2018)
- [Benchmarking Natural Language Understanding Services for building Conversational Agents](https://mlsanity.com/paper/1903.05566/) (2019)
- [S3FD: Single Shot Scale-invariant Face Detector](https://mlsanity.com/paper/1708.05237/) (2017)
- [UR-FUNNY: A Multimodal Language Dataset for Understanding Humor](https://mlsanity.com/paper/1904.06618/) (2019)
- [Is Someone Speaking? Exploring Long-term Temporal Features for Audio-visual Active Speaker Detection](https://mlsanity.com/paper/2107.06592/) (2021)
- [A Stack-Propagation Framework with Token-Level Intent Detection for Spoken Language Understanding](https://mlsanity.com/paper/1909.02188/) (2019)
- [AVA-ActiveSpeaker: An Audio-Visual Dataset for Active Speaker Detection](https://mlsanity.com/paper/1901.01342/) (2019)
- [RecipeQA: A Challenge Dataset for Multimodal Comprehension of Cooking Recipes](https://mlsanity.com/paper/1809.00812/) (2018)
- [Joint Slot Filling and Intent Detection via Capsule Neural Networks](https://mlsanity.com/paper/1812.09471/) (2018)

## Built on by (most cited first, 30 of 41)

- [UniMSE: Towards Unified Multimodal Sentiment Analysis and Emotion Recognition](https://mlsanity.com/paper/2211.11256/) (2022)
- [MIntRec2.0: A Large-scale Benchmark Dataset for Multimodal Intent Recognition and Out-of-scope Detection in Conversations](https://mlsanity.com/paper/2403.10943/) (2024)
- [Token-Level Contrastive Learning with Modality-Aware Prompting for Multimodal Intent Recognition](https://mlsanity.com/paper/2312.14667/) (2023)
- [Emotion and Intent Joint Understanding in Multimodal Conversation: A Benchmarking Dataset](https://mlsanity.com/paper/2407.02751/) (2024)
- [Speech-Text Dialog Pre-training for Spoken Dialog Understanding with Explicit Cross-Modal Alignment](https://mlsanity.com/paper/2305.11579/) (2023)
- [Toxic Memes: A Survey of Computational Perspectives on the Detection and Explanation of Meme Toxicities](https://mlsanity.com/paper/2406.07353/) (2024)
- [Recent Advances in Multimodal Affective Computing: An NLP Perspective](https://mlsanity.com/paper/2409.07388/) (2024)
- [A-MESS: Anchor based Multimodal Embedding with Semantic Synchronization for Multimodal Intent Recognition](https://mlsanity.com/paper/2503.19474/) (2025)
- [A Clustering Framework for Unsupervised and Semi-supervised New Intent Discovery](https://mlsanity.com/paper/2304.07699/) (2023)
- [Identifying User Goals from UI Trajectories](https://mlsanity.com/paper/2406.14314/) (2024)
- [Can Large Language Models Help Multimodal Language Analysis? MMLA: A Comprehensive Benchmark](https://mlsanity.com/paper/2504.16427/) (2025)
- [SDIF-DA: A Shallow-to-Deep Interaction Framework with Data Augmentation for Multi-modal Intent Detection](https://mlsanity.com/paper/2401.00424/) (2023)
- [Text Takes Over: A Study of Modality Bias in Multimodal Intent Detection](https://mlsanity.com/paper/2508.16122/) (2025)
- [Deep Learning Approaches for Multimodal Intent Recognition: A Survey](https://mlsanity.com/paper/2507.22934/) (2025)
- [Tri-Subspaces Disentanglement for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2602.19585/) (2026)
- [OmniOPSD: Rationale-Privileged On-Policy Self-Distillation for Affective Computing](https://mlsanity.com/paper/2606.15920/) (2026)
- [Multimodal Transformers are Hierarchical Modal-wise Heterogeneous Graphs](https://mlsanity.com/paper/2505.01068/) (2025)
- [Capsule Network-Based Semantic Intent Modeling for Human-Computer Interaction](https://mlsanity.com/paper/2507.00540/) (2025)
- [Unsupervised Multimodal Clustering for Semantics Discovery in Multimodal Utterances](https://mlsanity.com/paper/2405.12775/) (2024)
- [Looking Beyond the Obvious: A Survey on Abstract Concept Recognition for Video Understanding](https://mlsanity.com/paper/2508.20765/) (2025)
- [Complementarity-Supervised Spectral-Band Routing for Multimodal Emotion Recognition](https://mlsanity.com/paper/2603.13340/) (2026)
- [Group Cognition Learning: Making Everything Better Through Governed Two-Stage Agents Collaboration](https://mlsanity.com/paper/2605.00370/) (2026)
- [Impact of Stickers on Multimodal Sentiment and Intent in Social Media: A New Task, Dataset and Baseline](https://mlsanity.com/paper/2405.08427/) (2024)
- [WDMIR: Wavelet-Driven Multimodal Intent Recognition](https://mlsanity.com/paper/2506.10011/) (2025)
- [Training LLMs with Reinforcement Learning for Intent-Aware Personalized Question Answering](https://mlsanity.com/paper/2605.12645/) (2026)
- [TECO: Improving Multimodal Intent Recognition with Text Enhancement through Commonsense Knowledge Extraction](https://mlsanity.com/paper/2412.08529/) (2024)
- [MVCL-DAF++: Enhancing Multimodal Intent Recognition via Prototype-Aware Contrastive Alignment and Coarse-to-Fine Dynamic Attention Fusion](https://mlsanity.com/paper/2509.17446/) (2025)
- [Nano-EmoX: Unifying Multimodal Emotional Intelligence from Perception to Empathy](https://mlsanity.com/paper/2603.02123/) (2026)
- [ADMC: Attention-based Diffusion Model for Missing Modalities Feature Completion](https://mlsanity.com/paper/2507.05624/) (2025)
- [DyKen-Hyena: Dynamic Kernel Generation via Cross-Modal Attention for Multimodal Intent Recognition](https://mlsanity.com/paper/2509.09940/) (2025)

## Abstract (arXiv)

Multimodal intent recognition is a significant task for understanding human language in real-world multimodal scenes. Most existing intent recognition methods have limitations in leveraging the multimodal information due to the restrictions of the benchmark datasets with only text information. This paper introduces a novel dataset for multimodal intent recognition (MIntRec) to address this issue. It formulates coarse-grained and fine-grained intent taxonomies based on the data collected from the TV series Superstore. The dataset consists of 2,224 high-quality samples with text, video, and audio modalities and has multimodal annotations among twenty intent categories. Furthermore, we provide annotated bounding boxes of speakers in each video segment and achieve an automatic process for speaker annotation. MIntRec is helpful for researchers to mine relationships between different modalities to enhance the capability of intent recognition. We extract features from each modality and model cross-modal interactions by adapting three powerful multimodal fusion methods to build baselines. Extensive experiments show that employing the non-verbal modalities achieves substantial improvements compared with the text-only modality, demonstrating the effectiveness of using multimodal information for intent recognition. The gap between the best-performing methods and humans indicates the challenge and importance of this task for the community. The full dataset and codes are available for use at https://github.com/thuiar/MIntRec.