mlsanityData to 7 Oct 2026

Multimodal Learning using Optimal Transport for Sarcasm and Humor Detection

arXiv 2110.10949 ↗First posted 21 Oct 2021cs.CVcited by 36

Not on the map. Why?

Abstract (arXiv)

Multimodal learning is an emerging yet challenging research area. In this paper, we deal with multimodal sarcasm and humor detection from conversational videos and image-text pairs. Being a fleeting action, which is reflected across the modalities, sarcasm detection is challenging since large datasets are not available for this task in the literature. Therefore, we primarily focus on resource-constrained training, where the number of training samples is limited. To this end, we propose a novel multimodal learning system, MuLOT (Multimodal Learning using Optimal Transport), which utilizes self-attention to exploit intra-modal correspondence and optimal transport for cross-modal correspondence. Finally, the modalities are combined with multimodal attention fusion to capture the inter-dependencies across modalities. We test our approach for multimodal sarcasm and humor detection on three benchmark datasets - MUStARD (video, audio, text), UR-FUNNY (video, audio, text), MST (image, text) and obtain 2.1%, 1.54%, and 2.34% accuracy improvements over state-of-the-art.

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). Builds on 34 of its 68 references that are in the map. First posted on arXiv 21 Oct 2021.

Builds on

34 of 68 references.

References · 34

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  7. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  8. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  9. 2018Deep contextualized word representationscited by 2,285
  10. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  11. 2017The Kinetics Human Action Video Datasetcited by 1,894
  12. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
22 more
  1. 2019Multimodal Transformer for Unaligned Multimodal Language Sequencescited by 637
  2. 2017Tensor Fusion Network for Multimodal Sentiment Analysiscited by 429
  3. 2020MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysiscited by 262
  4. 2019Towards Multimodal Sarcasm Detection (An _Obviously_ Perfect Paper)cited by 173
  5. 2018Memory Fusion Network for Multi-view Sequential Learningcited by 167
  6. 2018Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalitiescited by 149
  7. 2019UR-FUNNY: A Multimodal Language Dataset for Understanding Humorcited by 130
  8. 2019Supervised Multimodal Bitransformers for Classifying Images and Textcited by 130
  9. 2018Words Can Shift: Dynamically Adjusting Word Representations Using Nonverbal Behaviorscited by 89
  10. 2021MOMENTA: A Multimodal Framework for Detecting Harmful Memes and Their Targetscited by 85
  11. 2019Learning Relationships between Text, Audio, and Video via Deep Canonical Correlation for Multimodal Language Analysiscited by 82
  12. 2021Detecting Harmful Memes and Their Targetscited by 77
  13. 2021Bi-Bimodal Modality Fusion for Correlation-Controlled Multimodal Sentiment Analysiscited by 45
  14. 2020Spatially Aware Multimodal Transformers for TextVQAcited by 44
  15. 2019Multi-task Learning for Multi-modal Emotion Recognition and Sentiment Analysiscited by 39
  16. 2021See, Hear, Read: Leveraging Multimodality with Guided Attention for Abstractive Text Summarizationcited by 18
  17. 2015We Are Humor Beings: Understanding and Predicting Visual Humorcited by 15
  18. 2019Complementary Fusion of Multi-Features and Multi-Modalities in Sentiment Analysiscited by 11
  19. 2021Exercise? I thought you said 'Extra Fries': Leveraging Sentence Demarcations and Multi-hop Attention for Meme Affect Analysiscited by 9
  20. 2018Representing Social Media Users for Sarcasm Detectioncited by 8
  21. 2019Joint Robust Voicing Detection and Pitch Estimation Based on Residual Harmonicscited by 6
  22. 2019Detection of Glottal Closure Instants from Speech Signals: a Quantitative Reviewcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 5; 2023: 9; 2024: 12; 2025: 4; 2026: 6.

36 papers.

Sort:

By area:

  1. 2023EgoVLPv2: Egocentric Video-Language Pre-training with Fusion in the Backbonecited by 130
  2. 2022Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questionscited by 116
  3. 2022Where in the World is this Image? Transformer-based Geo-localization in the Wildcited by 58
  4. 2023Let's Think Outside the Box: Exploring Leap-of-Thought in Large Language Models with Creative Humor Generationcited by 56
  5. 2024Multimodal Prototyping for cancer survival predictioncited by 47
  6. 2023DialogueLLM: Context and Emotion Knowledge-Tuned Large Language Models for Emotion Recognition in Conversationscited by 29
  7. 2023SMILE: Multimodal Dataset for Understanding Laughter in Video with Language Modelscited by 28
  8. 2022COLD Fusion: Calibrated and Ordinal Latent Distribution Fusion for Uncertainty-Aware Multimodal Emotion Recognitioncited by 22
  9. 2022VoLTA: Vision-Language Transformer with Weakly-Supervised Local-Feature Alignmentcited by 20
  10. 2023Multimodal Learning Without Labeled Multimodal Data: Guarantees and Applicationscited by 20
  11. 2024A Survey of Multimodal Sarcasm Detectioncited by 19
  12. 2024Encoding and Controlling Global Semantics for Long-form Video Question Answeringcited by 8
24 more
  1. 2025Can Large Language Models Help Multimodal Language Analysis? MMLA: A Comprehensive Benchmarkcited by 8
  2. 2024Mixture-of-Prompt-Experts for Multi-modal Semantic Understandingcited by 6
  3. 2024Efficient Feature Extraction and Late Fusion Strategy for Audiovisual Emotional Mimicry Intensity Estimationcited by 6
  4. 2023Context-aware attention layers coupled with optimal transport domain adaptation and multimodal fusion methods for recognizing dementia from spontaneous speechcited by 5
  5. 2023LICO: Explainable Models with Language-Image Consistencycited by 5
  6. 2025MUStReason: A Benchmark for Diagnosing Pragmatic Reasoning in Video-LMs for Multimodal Sarcasm Detectioncited by 5
  7. 2022Towards Multimodal Prediction of Spontaneous Humour: A Novel Dataset and First Resultscited by 4
  8. 2024End-to-end Semantic-centric Video-based Multimodal Affective Computingcited by 4
  9. 2025Spoken in Jest, Detected in Earnest: A Systematic Review of Sarcasm Recognition -- Multimodal Fusion, Challenges, and Future Prospectscited by 4
  10. 2024The MuSe 2024 Multimodal Sentiment Analysis Challenge: Social Perception and Humor Recognitioncited by 3
  11. 2023TOT: Topology-Aware Optimal Transport For Multimodal Hate Detectioncited by 2
  12. 2024Modelling Visual Semantics via Image Captioning to extract Enhanced Multi-Level Cross-Modal Semantic Incongruity Representation with Attention for Multimodal Sarcasm Detectioncited by 2
  13. 2025Evaluating Multimodal Large Language Models on Spoken Sarcasm Understandingcited by 2
  14. 2026I Came, I Saw, I Explained: Benchmarking Multimodal LLMs on Figurative Meaning in Memescited by 2
  15. 2023Cross-Attention is Not Enough: Incongruity-Aware Dynamic Hierarchical Fusion for Multimodal Affect Recognitioncited by 1
  16. 2024Alignment Helps Make the Most of Multimodal Datacited by 1
  17. 2024COT: A Generative Approach for Hate Speech Counter-Narratives via Contrastive Optimal Transportcited by 1
  18. 2024NYK-MS: A Well-annotated Multi-modal Metaphor and Sarcasm Understanding Benchmark on Cartoon-Caption Datasetcited by 1
  19. 2026CaReFlow: Cyclic Adaptive Rectified Flow for Multimodal Fusioncited by 1
  20. 2024An Eye for an Ear: Zero-shot Audio Description Leveraging an Image Captioner using Audiovisual Distribution Alignmentcited by 0
  21. 2026A Vision for Multisensory Intelligence: Sensing, Science, and Synergycited by 0
  22. 2026Multimodal Alignment Through Joint Kernel Entropic Gromov--Wasserstein Optimal Transportcited by 0
  23. 2026Mitigating Strong-Modality Collapse in Multimodal Learning via Inverted Asymmetric Fusioncited by 0
  24. 2026Reliability-aware Cross-sample Enhancement for Robust Multimodal Sentiment Analysiscited by 0