# Multi-Granular Multimodal Clue Fusion for Meme Understanding

arXiv: https://arxiv.org/abs/2503.12560 · HTML: https://mlsanity.com/paper/2503.12560/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 42 references that are in the map.
- First posted on arXiv 16 Mar 2025.

## Builds on (most cited first, 26 of 26)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [NExT-GPT: Any-to-Any Multimodal LLM](https://mlsanity.com/paper/2309.05519/) (2023)
- [The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes](https://mlsanity.com/paper/2005.04790/) (2020)
- [Rethinking Spatial Dimensions of Vision Transformers](https://mlsanity.com/paper/2103.16302/) (2021)
- [Cross-Lingual Ability of Multilingual BERT: An Empirical Study](https://mlsanity.com/paper/1912.07840/) (2019)
- [Video-of-Thought: Step-by-Step Video Reasoning from Perception to Cognition](https://mlsanity.com/paper/2501.03230/) (2024)
- [Fine-Grained Semantically Aligned Vision-Language Pre-Training](https://mlsanity.com/paper/2208.02515/) (2022)
- [Compositional Temporal Grounding with Structured Variational Cross-Graph Correspondence Learning](https://mlsanity.com/paper/2203.13049/) (2022)
- [Improving Image Captioning by Leveraging Intra- and Inter-layer Global Representation in Transformer Network](https://mlsanity.com/paper/2012.07061/) (2020)
- [Towards Semantic Equivalence of Tokenization in Multimodal LLM](https://mlsanity.com/paper/2406.05127/) (2024)
- [You Can Ground Earlier than See: An Effective and Efficient Pipeline for Temporal Sentence Grounding in Compressed Videos](https://mlsanity.com/paper/2303.07863/) (2023)
- [Decoding the Underlying Meaning of Multimodal Hateful Memes](https://mlsanity.com/paper/2305.17678/) (2023)
- [Enhancing Video-Language Representations with Structural Spatio-Temporal Alignment](https://mlsanity.com/paper/2406.19255/) (2024)
- [On the Evolution of (Hateful) Memes by Means of Multimodal Contrastive Learning](https://mlsanity.com/paper/2212.06573/) (2022)
- [Your Data Is Not Perfect: Towards Cross-Domain Out-of-Distribution Detection in Class-Imbalanced Data](https://mlsanity.com/paper/2412.06284/) (2024)
- [Information Screening whilst Exploiting! Multimodal Relation Extraction with Feature Denoising and Multimodal Topic Modeling](https://mlsanity.com/paper/2305.11719/) (2023)
- [Memes in the Wild: Assessing the Generalizability of the Hateful Memes Challenge Dataset](https://mlsanity.com/paper/2107.04313/) (2021)
- [Metaphor Detection via Explicit Basic Meanings Modelling](https://mlsanity.com/paper/2305.17268/) (2023)
- [Adversarial Multi-task Learning for End-to-end Metaphor Detection](https://mlsanity.com/paper/2305.16638/) (2023)
- [Reverse Multi-Choice Dialogue Commonsense Inference with Graph-of-Thought](https://mlsanity.com/paper/2312.15291/) (2023)
- [ECQED: Emotion-Cause Quadruple Extraction in Dialogs](https://mlsanity.com/paper/2306.03969/) (2023)
- [A Bi-directional Multi-hop Inference Model for Joint Dialog Sentiment Classification and Act Recognition](https://mlsanity.com/paper/2308.04424/) (2023)

## Built on by (most cited first, 14 of 14)

- [Zero-Shot Conversational Stance Detection: Dataset and Approaches](https://mlsanity.com/paper/2506.17693/) (2025)
- [TrimTokenator: Towards Adaptive Visual Token Pruning for Large Multimodal Models](https://mlsanity.com/paper/2509.00320/) (2025)
- [Enhancing Hyperbole and Metaphor Detection with Their Bidirectional Dynamic Interaction and Emotion Knowledge](https://mlsanity.com/paper/2506.15504/) (2025)
- [MGHFT: Multi-Granularity Hierarchical Fusion Transformer for Cross-Modal Sticker Emotion Recognition](https://mlsanity.com/paper/2507.18929/) (2025)
- [Concept Drift Guided LayerNorm Tuning for Efficient Multimodal Metaphor Identification](https://mlsanity.com/paper/2505.11237/) (2025)
- [DALR: Dual-level Alignment Learning for Multimodal Sentence Representation Learning](https://mlsanity.com/paper/2506.21096/) (2025)
- [Structured Prompting and LLM Ensembling for Multimodal Conversational Aspect-based Sentiment Analysis](https://mlsanity.com/paper/2512.22603/) (2025)
- [Dynamic Emotion and Personality Profiling for Multimodal Deception Detection](https://mlsanity.com/paper/2604.17037/) (2026)
- [Beyond Translation: Cross-Cultural Meme Transcreation with Vision-Language Models](https://mlsanity.com/paper/2602.02510/) (2026)
- [Are Emotion and Rhetoric Neurons in LLM? Neuron Recognition and Adaptive Masking for Emotion-Rhetoric Prediction Steering](https://mlsanity.com/paper/2604.17255/) (2026)
- [MetaphorVU: Towards Metaphorical Video Understanding](https://mlsanity.com/paper/2605.25461/) (2026)
- [Beyond the Literal: Decomposing Pragmatic Intent in Multimodal Meme Understanding](https://mlsanity.com/paper/2606.03604/) (2026)
- [Towards High-Level Semantic Intelligence](https://mlsanity.com/paper/2607.24082/) (2026)
- [MMDS-Bench: Benchmarking Multimodal Large Language Models on Dynamic Stance in Social Media Interactions](https://mlsanity.com/paper/2608.30903/) (2026)

## Abstract (arXiv)

With the continuous emergence of various social media platforms frequently used in daily life, the multimodal meme understanding (MMU) task has been garnering increasing attention. MMU aims to explore and comprehend the meanings of memes from various perspectives by performing tasks such as metaphor recognition, sentiment analysis, intention detection, and offensiveness detection. Despite making progress, limitations persist due to the loss of fine-grained metaphorical visual clue and the neglect of multimodal text-image weak correlation. To overcome these limitations, we propose a multi-granular multimodal clue fusion model (MGMCF) to advance MMU. Firstly, we design an object-level semantic mining module to extract object-level image feature clues, achieving fine-grained feature clue extraction and enhancing the model's ability to capture metaphorical details and semantics. Secondly, we propose a brand-new global-local cross-modal interaction model to address the weak correlation between text and images. This model facilitates effective interaction between global multimodal contextual clues and local unimodal feature clues, strengthening their representations through a bidirectional cross-modal attention mechanism. Finally, we devise a dual-semantic guided training strategy to enhance the model's understanding and alignment of multimodal representations in the semantic space. Experiments conducted on the widely-used MET-MEME bilingual dataset demonstrate significant improvements over state-of-the-art baselines. Specifically, there is an 8.14% increase in precision for offensiveness detection task, and respective accuracy enhancements of 3.53%, 3.89%, and 3.52% for metaphor recognition, sentiment analysis, and intention detection tasks. These results, underpinned by in-depth analyses, underscore the effectiveness and potential of our approach for advancing MMU.