MemeCULT-1K: Benchmarking South Asian Cultural Context and Humor Understanding of Multimodal Models
On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning
Abstract (arXiv)
Meme understanding goes beyond recognizing visual content or literal text; it requires implicit cultural knowledge and pragmatic inference that most vision-language models still lack. We introduce MemeCULT-1K, a multilingual benchmark of 1,000 South Asian memes in Bengali, English, and Hindi, where each meme is paired with a cultural context note and three human-written explanations, along with a supplementary set of 54 Bengali regional dialect memes. We evaluate thirteen popular Vision Language Models (VLMs) under two settings: meme-only and context-aware. Providing minimal cultural context yields consistent gains across all models and languages: mean SBERT similarity improves from 44.6 to 56.4 (+11.8), BLEURT from 37.3 to 42.3 (+5.0), and LLM-as-a-Judge scores from 2.57 to 3.43 out of 5 (+0.86). Fine-grained error analysis reveals that closed-source models fail mainly on entity and reference misidentification, while open-source models are bottlenecked by broader cultural knowledge gaps, with linguistic and phonological failures proving the most context-resistant across both. These results highlight the difficulty of culturally grounded meme understanding and motivate future work on explicit cultural knowledge integration. Our dataset and code are publicly available at TawsifDipto17/MemeCULT-1K.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 14 of its 16 references that are in the map. First posted on arXiv 1 Sep 2026.
Builds on
14 of 16 references.
References · 14
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
- 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
- 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
- 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,203
- 2020The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memescited by 490
- 2020SemEval-2020 Task 8: Memotion Analysis -- The Visuo-Lingual Metaphor!cited by 78
- 2023MemeCap: A Dataset for Captioning and Interpreting Memescited by 46
- 2023BanglaAbuseMeme: A Dataset for Bengali Abusive Meme Classificationcited by 18
- 2024See It from My Perspective: How Language Affects Cultural Bias in Image Understandingcited by 17
- 2025Are Vision-Language Models Safe in the Wild? A Meme-Based Benchmark Studycited by 12
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026BanglaMemeX: Advancing Cultural Metaphoric Image Interpretation in Bangla with a Multimodal Explainable Dataset
- 2026Beyond Translation: Cross-Cultural Meme Transcreation with Vision-Language Models
- 2025MemeReaCon: Probing Contextual Meme Understanding in Large Vision-Language Models
- 2026MemeBench: What LVLMs Miss When Interpreting Culture-Dependent Memes
- 2025Large Vision-Language Models for Knowledge-Grounded Data Annotation of Memes
- 2026MemeLens: Multilingual Multitask VLMs for Memes
- 2026Many Dialects, Many Languages, One Cultural Lens: Evaluating Multilingual VLMs for Bengali Culture Understanding Across Historically Linked Languages and Regional Dialects
- 2026From Native Memes to Global Moderation: Cross-Cultural Evaluation of Vision-Language Models for Hateful Meme Detection