# Decoding Memes: Benchmarking Narrative Role Classification across Multilingual and Multimodal Models

arXiv: https://arxiv.org/abs/2506.23122 · HTML: https://mlsanity.com/paper/2506.23122/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 29 of its 50 references that are in the map.
- First posted on arXiv 29 Jun 2025.

## Builds on (most cited first, 29 of 29)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing](https://mlsanity.com/paper/2111.09543/) (2021)
- [The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes](https://mlsanity.com/paper/2005.04790/) (2020)
- [HateXplain: A Benchmark Dataset for Explainable Hate Speech Detection](https://mlsanity.com/paper/2012.10289/) (2020)
- [Social Bias Frames: Reasoning about Social and Power Implications of Language](https://mlsanity.com/paper/1911.03891/) (2019)
- [MuRIL: Multilingual Representations for Indian Languages](https://mlsanity.com/paper/2103.10730/) (2021)
- [Multilingual and Multi-Aspect Hate Speech Analysis](https://mlsanity.com/paper/1908.11049/) (2019)
- [Exploring Hate Speech Detection in Multimodal Publications](https://mlsanity.com/paper/1910.03814/) (2019)
- [GLUECoS : An Evaluation Benchmark for Code-Switched NLP](https://mlsanity.com/paper/2004.12376/) (2020)
- [MOMENTA: A Multimodal Framework for Detecting Harmful Memes and Their Targets](https://mlsanity.com/paper/2109.05184/) (2021)
- [SemEval-2020 Task 8: Memotion Analysis -- The Visuo-Lingual Metaphor!](https://mlsanity.com/paper/2008.03781/) (2020)
- [Detecting Harmful Memes and Their Targets](https://mlsanity.com/paper/2110.00413/) (2021)
- [Detecting and Understanding Harmful Memes: A Survey](https://mlsanity.com/paper/2205.04274/) (2022)
- [On the Origins of Memes by Means of Fringe Web Communities](https://mlsanity.com/paper/1805.12512/) (2018)
- [Anyone Can Become a Troll: Causes of Trolling Behavior in Online Discussions](https://mlsanity.com/paper/1702.01119/) (2017)
- [Multimodal Learning for Hateful Memes Detection](https://mlsanity.com/paper/2011.12870/) (2020)
- [Recent Advances in Hate Speech Moderation: Multimodality and the Role of Large Models](https://mlsanity.com/paper/2401.16727/) (2024)
- [Data Bootstrapping Approaches to Improve Low Resource Abusive Language Detection for Indic Languages](https://mlsanity.com/paper/2204.12543/) (2022)
- [DISARM: Detecting the Victims Targeted by Harmful Memes](https://mlsanity.com/paper/2205.05738/) (2022)
- [#MeTooMA: Multi-Aspect Annotations of Tweets Related to the MeToo Movement](https://mlsanity.com/paper/1912.06927/) (2019)
- [Characterizing the Entities in Harmful Memes: Who is the Hero, the Villain, the Victim?](https://mlsanity.com/paper/2301.11219/) (2023)
- [MemeMQA: Multimodal Question Answering for Memes via Rationale-Based Inferencing](https://mlsanity.com/paper/2405.11215/) (2024)
- [MEMEX: Detecting Explanatory Evidence for Memes via Knowledge-Enriched Contextualization](https://mlsanity.com/paper/2305.15913/) (2023)
- [Emotion-Aware Multimodal Fusion for Meme Emotion Detection](https://mlsanity.com/paper/2403.10279/) (2024)
- [Race, Ethnicity and National Origin-based Discrimination in Social Media and Hate Crimes Across 100 U.S. Cities](https://mlsanity.com/paper/1902.00119/) (2019)

## Abstract (arXiv)

This work investigates the challenging task of identifying narrative roles - Hero, Villain, Victim, and Other - in Internet memes, across three diverse test sets spanning English and code-mixed (English-Hindi) languages. Building on an annotated dataset originally skewed toward the 'Other' class, we explore a more balanced and linguistically diverse extension, originally introduced as part of the CLEF 2024 shared task. Comprehensive lexical and structural analyses highlight the nuanced, culture-specific, and context-rich language used in real memes, in contrast to synthetically curated hateful content, which exhibits explicit and repetitive lexical markers. To benchmark the role detection task, we evaluate a wide spectrum of models, including fine-tuned multilingual transformers, sentiment and abuse-aware classifiers, instruction-tuned LLMs, and multimodal vision-language models. Performance is assessed under zero-shot settings using precision, recall, and F1 metrics. While larger models like DeBERTa-v3 and Qwen2.5-VL demonstrate notable gains, results reveal consistent challenges in reliably identifying the 'Victim' class and generalising across cultural and code-mixed content. We also explore prompt design strategies to guide multimodal models and find that hybrid prompts incorporating structured instructions and role definitions offer marginal yet consistent improvements. Our findings underscore the importance of cultural grounding, prompt engineering, and multimodal reasoning in modelling subtle narrative framings in visual-textual content.