# Detecting Offensive Memes with Social Biases in Singapore Context Using Multimodal Large Language Models

arXiv: https://arxiv.org/abs/2502.18101 · HTML: https://mlsanity.com/paper/2502.18101/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 13 of its 48 references that are in the map.
- First posted on arXiv 25 Feb 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Document understanding and multimodal RAG](https://mlsanity.com/mllm/document-understanding-and-multimodal-rag/)

## Builds on (most cited first, 13 of 13)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [DoRA: Weight-Decomposed Low-Rank Adaptation](https://mlsanity.com/paper/2402.09353/) (2024)
- [The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes](https://mlsanity.com/paper/2005.04790/) (2020)
- [LoRA+: Efficient Low Rank Adaptation of Large Models](https://mlsanity.com/paper/2402.12354/) (2024)
- [A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA](https://mlsanity.com/paper/2312.03732/) (2023)
- [PP-OCRv3: More Attempts for the Improvement of Ultra Lightweight OCR System](https://mlsanity.com/paper/2206.03001/) (2022)
- [Disentangling Hate in Online Memes](https://mlsanity.com/paper/2108.06207/) (2021)
- [MemeCap: A Dataset for Captioning and Interpreting Memes](https://mlsanity.com/paper/2305.13703/) (2023)
- [Decoding the Underlying Meaning of Multimodal Hateful Memes](https://mlsanity.com/paper/2305.17678/) (2023)
- [Benchmark dataset of memes with text transcriptions for automatic detection of multi-modal misogynistic content](https://mlsanity.com/paper/2106.08409/) (2021)
- [LionGuard: Building a Contextualized Moderation Classifier to Tackle Localized Unsafe Content](https://mlsanity.com/paper/2407.10995/) (2024)

## Abstract (arXiv)

Traditional online content moderation systems struggle to classify modern multimodal means of communication, such as memes, a highly nuanced and information-dense medium. This task is especially hard in a culturally diverse society like Singapore, where low-resource languages are used and extensive knowledge on local context is needed to interpret online content. We curate a large collection of 112K memes labeled by GPT-4V for fine-tuning a VLM to classify offensive memes in Singapore context. We show the effectiveness of fine-tuned VLMs on our dataset, and propose a pipeline containing OCR, translation and a 7-billion parameter-class VLM. Our solutions reach 80.62% accuracy and 0.8192 AUROC on a held-out test set, and can greatly aid human in moderating online contents. The dataset, code, and model weights have been open-sourced at https://github.com/aliencaocao/vlm-for-memes-aisg.