# InterCLIP-MEP: Interactive CLIP and Memory-Enhanced Predictor for Multi-modal Sarcasm Detection

arXiv: https://arxiv.org/abs/2406.16464 · HTML: https://mlsanity.com/paper/2406.16464/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 68 references that are in the map.
- First posted on arXiv 24 Jun 2024.

## Builds on (most cited first, 27 of 27)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision](https://mlsanity.com/paper/2102.03334/) (2021)
- [Convolutional Neural Networks for Sentence Classification](https://mlsanity.com/paper/1408.5882/) (2014)
- [An Empirical Study of Example Forgetting during Deep Neural Network Learning](https://mlsanity.com/paper/1812.05159/) (2018)
- [Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP](https://mlsanity.com/paper/2210.04150/) (2022)
- [Memory Networks](https://mlsanity.com/paper/1410.3916/) (2014)
- [Detecting Sarcasm in Multimodal Social Platforms](https://mlsanity.com/paper/1608.02289/) (2016)
- [Reasoning with Sarcasm by Reading In-between](https://mlsanity.com/paper/1805.02856/) (2018)
- [A Deeper Look into Sarcastic Tweets Using Deep Convolutional Neural Networks](https://mlsanity.com/paper/1610.08815/) (2016)
- [Question Aware Vision Transformer for Multimodal Reasoning](https://mlsanity.com/paper/2402.05472/) (2024)
- [MMSD2.0: Towards a Reliable Multi-modal Sarcasm Detection System](https://mlsanity.com/paper/2307.07135/) (2023)
- [Towards Multi-Modal Sarcasm Detection via Hierarchical Congruity Modeling with Knowledge Enhancement](https://mlsanity.com/paper/2210.03501/) (2022)
- [Modelling Context with User Embeddings for Sarcasm Detection in Social Media](https://mlsanity.com/paper/1607.00976/) (2016)
- [Adapting CLIP For Phrase Localization Without Further Training](https://mlsanity.com/paper/2204.03647/) (2022)
- [Multi-source Semantic Graph-based Multimodal Sarcasm Explanation Generation](https://mlsanity.com/paper/2306.16650/) (2023)
- [Seeing in Flowing: Adapting CLIP for Action Recognition with Motion Prompts Learning](https://mlsanity.com/paper/2308.04828/) (2023)
- [Are Word Embedding-based Features Useful for Sarcasm Detection?](https://mlsanity.com/paper/1610.00883/) (2016)
- [A Quantum Probability Driven Framework for Joint Multi-Modal Sarcasm, Sentiment and Emotion Analysis](https://mlsanity.com/paper/2306.03650/) (2023)
- [DocMSU: A Comprehensive Benchmark for Document-level Multimodal Sarcasm Understanding](https://mlsanity.com/paper/2312.16023/) (2023)
- [NTUA-SLP at SemEval-2018 Task 3: Tracking Ironic Tweets using Ensembles of Word and Character Level Attentive RNNs](https://mlsanity.com/paper/1804.06659/) (2018)

## Built on by (most cited first, 4 of 4)

- [RCLMuFN: Relational Context Learning and Multiplex Fusion Network for Multimodal Sarcasm Detection](https://mlsanity.com/paper/2412.13008/) (2024)
- [Seeing Sarcasm Through Different Eyes: Analyzing Multimodal Sarcasm Perception in Large Vision-Language Models](https://mlsanity.com/paper/2503.12149/) (2025)
- [Training-Free Multimodal Deepfake Detection via Graph Reasoning](https://mlsanity.com/paper/2509.21774/) (2025)
- [Dynamic Gated Cross-Modal Fusion with Sarcastic-aware Contrastive Regularization for Multimodal Sarcasm Detection](https://mlsanity.com/paper/2608.19942/) (2026)

## Abstract (arXiv)

Sarcasm in social media, frequently conveyed through the interplay of text and images, presents significant challenges for sentiment analysis and intention mining. Existing multi-modal sarcasm detection approaches have been shown to excessively depend on superficial cues within the textual modality, exhibiting limited capability to accurately discern sarcasm through subtle text-image interactions. To address this limitation, a novel framework, InterCLIP-MEP, is proposed. This framework integrates Interactive CLIP (InterCLIP), which employs an efficient training strategy to derive enriched cross-modal representations by embedding inter-modal information directly into each encoder, while using approximately 20.6× fewer trainable parameters compared with existing state-of-the-art (SOTA) methods. Furthermore, a Memory-Enhanced Predictor (MEP) is introduced, featuring a dynamic dual-channel memory mechanism that captures and retains valuable knowledge from test samples during inference, serving as a non-parametric classifier to enhance sarcasm detection robustness. Extensive experiments on MMSD, MMSD2.0, and DocMSU show that InterCLIP-MEP achieves SOTA performance, specifically improving accuracy by 1.08% and F1 score by 1.51% on MMSD2.0. Under distributional shift evaluation, it attains 73.96% accuracy, exceeding its memory-free variant by nearly 10% and the previous SOTA by over 15%, demonstrating superior stability and adaptability. The implementation of InterCLIP-MEP is publicly available at https://github.com/CoderChen01/InterCLIP-MEP.