mlsanityData to 14 Oct 2026

On VLMs for Diverse Tasks in Multimodal Meme Classification

arXiv 2505.20937 ↗First posted 27 May 2025cs.CLcited by 2

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

In this paper, we present a comprehensive and systematic analysis of vision-language models (VLMs) for disparate meme classification tasks. We introduced a novel approach that generates a VLM-based understanding of meme images and fine-tunes the LLMs on textual understanding of the embedded meme text for improving the performance. Our contributions are threefold: (1) Benchmarking VLMs with diverse prompting strategies purposely to each sub-task; (2) Evaluating LoRA fine-tuning across all VLM components to assess performance gains; and (3) Proposing a novel approach where detailed meme interpretations generated by VLMs are used to train smaller language models (LLMs), significantly improving classification. The strategy of combining VLMs with LLMs improved the baseline performance by 8.34%, 3.52% and 26.24% for sarcasm, offensive and sentiment classification, respectively. Our results reveal the strengths and limitations of VLMs and present a novel strategy for meme understanding.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 31 of its 68 references that are in the map. First posted on arXiv 27 May 2025.

Builds on

31 of 68 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution18 Sep
  2. 2024The Revolution of Multimodal Large Language Models: A Survey19 Feb
  3. 2023Improved Baselines with Visual Instruction Tuning5 Oct

Other references · 28

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  8. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  9. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  10. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  11. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
  12. 2023A Survey on Evaluation of Large Language Modelscited by 1,242
16 more
  1. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
  2. 2023Vision-Language Models for Vision Tasks: A Surveycited by 839
  3. 2017Adversarial Examples for Evaluating Reading Comprehension Systemscited by 571
  4. 2018Grounding Visual Explanationscited by 93
  5. 2021MOMENTA: A Multimodal Framework for Detecting Harmful Memes and Their Targetscited by 86
  6. 2020SemEval-2020 Task 8: Memotion Analysis -- The Visuo-Lingual Metaphor!cited by 78
  7. 2023Prompting for Multimodal Hateful Meme Classificationcited by 69
  8. 2024MemeCLIP: Leveraging CLIP Representations for Multimodal Meme Classificationcited by 30
  9. 2022DISARM: Detecting the Victims Targeted by Harmful Memescited by 20
  10. 2024MemeGuard: An LLM and VLM-based Framework for Advancing Content Moderation via Meme Interventioncited by 20
  11. 2022What do you MEME? Generating Explanations for Visual Semantic Role Labelling in Memescited by 18
  12. 2020A Multimodal Memes Classification: A Survey and Open Research Issuescited by 14
  13. 2021Caption Enriched Samples for Improving Hateful Memes Detectioncited by 14
  14. 2021Analyzing Online Political Advertisementscited by 11
  15. 2023Leveraging Large Language Models for Scalable Vector Graphics-Driven Image Understandingcited by 10
  16. 2024M3Hop-CoT: Misogynous Meme Identification with Multimodal Multi-hop Chain-of-Thoughtcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026MemeLens: Multilingual Multitask VLMs for Memescited by 6
  2. 2026Beyond Accuracy: A Qualitative Analysis of Vision-Language Models for Hate Speech Detection in Memescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Detecting and Correcting Hate Speech in Multimodal Memes with Large Visual Language Model
  2. 2025Detecting Offensive Memes with Social Biases in Singapore Context Using Multimodal Large Language Models
  3. 2025Demystifying Hateful Content: Leveraging Large Multimodal Models for Hateful Meme Detection with Explainable Decisions
  4. 2025Detecting and Mitigating Hateful Content in Multimodal Memes with Vision-Language Models
  5. 2023A Review of Vision-Language Models and their Performance on the Hateful Memes Challenge
  6. 2024Exploring the Limits of Zero Shot Vision Language Models for Hate Meme Detection: The Vulnerabilities and their Interpretations
  7. 2026Beyond a Joke: Multi-Angle Reasoning for Detecting and Explaining Harmful Humor in Memes
  8. 2025Large Vision-Language Models for Knowledge-Grounded Data Annotation of Memes