mlsanityData to 14 Oct 2026

SDIF-DA: A Shallow-to-Deep Interaction Framework with Data Augmentation for Multi-modal Intent Detection

arXiv 2401.00424 ↗First posted 31 Dec 2023cs.CLcited by 7

Not on the map. Why?

Abstract (arXiv)

Multi-modal intent detection aims to utilize various modalities to understand the user's intentions, which is essential for the deployment of dialogue systems in real-world scenarios. The two core challenges for multi-modal intent detection are (1) how to effectively align and fuse different features of modalities and (2) the limited labeled multi-modal intent training data. In this work, we introduce a shallow-to-deep interaction framework with data augmentation (SDIF-DA) to address the above challenges. Firstly, SDIF-DA leverages a shallow-to-deep interaction module to progressively and effectively align and fuse features across text, video, and audio modalities. Secondly, we propose a ChatGPT-based data augmentation approach to automatically augment sufficient training data. Experimental results demonstrate that SDIF-DA can effectively align and fuse multi-modal features by achieving state-of-the-art performance. In addition, extensive analyses show that the introduced data augmentation approach can successfully distill knowledge from the large language model.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 15 of its 23 references that are in the map. First posted on arXiv 31 Dec 2023.

Builds on

15 of 23 references.

References · 15

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  4. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  5. 2019Multimodal Transformer for Unaligned Multimodal Language Sequencescited by 645
  6. 2023A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPTcited by 368
  7. 2020MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysiscited by 265
  8. 2023AugGPT: Leveraging ChatGPT for Text Data Augmentationcited by 130
  9. 2019A Stack-Propagation Framework with Token-Level Intent Detection for Spoken Language Understandingcited by 107
  10. 2021A Survey on Spoken Language Understanding: Recent Advances and New Frontierscited by 53
  11. 2021Bi-Bimodal Modality Fusion for Correlation-Controlled Multimodal Sentiment Analysiscited by 45
  12. 2022MIntRec: A New Dataset for Multimodal Intent Recognitioncited by 41
3 more
  1. 2020Leveraging Unpaired Text Data for Training End-to-End Speech-to-Intent Systemscited by 38
  2. 2019Integrating Text and Image: Determining Multimodal Document Intent in Instagram Postscited by 37
  3. 2020Tie Your Embeddings Down: Cross-Modal Latent Spaces for End-to-end Spoken Language Understandingcited by 23

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 4; 2026: 1.

7 papers.

  1. 2024Meerkat: Audio-Visual Large Language Model for Grounding in Space and Timecited by 26
  2. 2025Text Takes Over: A Study of Modality Bias in Multimodal Intent Detectioncited by 7
  3. 2025Deep Learning Approaches for Multimodal Intent Recognition: A Surveycited by 6
  4. 2025WDMIR: Wavelet-Driven Multimodal Intent Recognitioncited by 2
  5. 2024TECO: Improving Multimodal Intent Recognition with Text Enhancement through Commonsense Knowledge Extractioncited by 1
  6. 2025IDALC: A Semi-Supervised Framework for Intent Detection and Active Learning based Correctioncited by 0
  7. 2026Modality Agreement- and Conflict-Aware Prototype Hypergraph Learning for Multimodal Intent Understandingcited by 0