# Towards Low-Resource Harmful Meme Detection with LMM Agents

arXiv: https://arxiv.org/abs/2411.05383 · HTML: https://mlsanity.com/paper/2411.05383/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 54 of its 73 references that are in the map.
- First posted on arXiv 8 Nov 2024.

## Builds on (most cited first, 30 of 54)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [REALM: Retrieval-Augmented Language Model Pre-Training](https://mlsanity.com/paper/2002.08909/) (2020)
- [Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection](https://mlsanity.com/paper/2310.11511/) (2023)
- [MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework](https://mlsanity.com/paper/2308.00352/) (2023)
- [What Makes Good In-Context Examples for GPT-3?](https://mlsanity.com/paper/2101.06804/) (2021)
- [Teaching Large Language Models to Self-Debug](https://mlsanity.com/paper/2304.05128/) (2023)
- [Atlas: Few-shot Learning with Retrieval Augmented Language Models](https://mlsanity.com/paper/2208.03299/) (2022)
- [ExpeL: LLM Agents Are Experiential Learners](https://mlsanity.com/paper/2308.10144/) (2023)
- [WizardCoder: Empowering Code Large Language Models with Evol-Instruct](https://mlsanity.com/paper/2306.08568/) (2023)
- [GPT-NeoX-20B: An Open-Source Autoregressive Language Model](https://mlsanity.com/paper/2204.06745/) (2022)
- [In-Context Retrieval-Augmented Language Models](https://mlsanity.com/paper/2302.00083/) (2023)
- [The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)](https://mlsanity.com/paper/2309.17421/) (2023)
- [Enabling Large Language Models to Generate Text with Citations](https://mlsanity.com/paper/2305.14627/) (2023)
- [The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes](https://mlsanity.com/paper/2005.04790/) (2020)

## Built on by (most cited first, 24 of 24)

- [MIND: A Multi-agent Framework for Zero-shot Harmful Meme Detection](https://mlsanity.com/paper/2507.06908/) (2025)
- [Robust Adaptation of Large Multimodal Models for Retrieval Augmented Hateful Meme Detection](https://mlsanity.com/paper/2502.13061/) (2025)
- [ExPO-HM: Learning to Explain-then-Detect for Hateful Meme Detection](https://mlsanity.com/paper/2510.08630/) (2025)
- [Read as You See: Guiding Unimodal LLMs for Low-Resource Explainable Harmful Meme Detection](https://mlsanity.com/paper/2506.08477/) (2025)
- [MemeReaCon: Probing Contextual Meme Understanding in Large Vision-Language Models](https://mlsanity.com/paper/2505.17433/) (2025)
- [From Recognition to Reasoning: Advancing Multimodal Harmful Meme Detection via Chain-of-Thought Alignment](https://mlsanity.com/paper/2506.18919/) (2025)
- [PEST: Parameter Efficient Steering of Blackbox VLMs via Agentic Few-shot Alignment for Hateful Meme Moderation](https://mlsanity.com/paper/2601.04692/) (2026)
- [What Media Frames Reveal About Stance: A Dataset and Study about Memes in Climate Change Discourse](https://mlsanity.com/paper/2505.16592/) (2025)
- [Yes FLoReNce, I Will Do Better Next Time! Agentic Feedback Reasoning for Humorous Meme Detection](https://mlsanity.com/paper/2601.07232/) (2026)
- [Multi-Agent Synergy-Driven Iterative Visual Narrative Synthesis](https://mlsanity.com/paper/2507.13285/) (2025)
- [MM-CRITIC: A Holistic Evaluation of Large Multimodal Models as Multimodal Critique](https://mlsanity.com/paper/2511.09067/) (2025)
- [Robust Harmful Meme Detection under Missing Modalities via Shared Representation Learning](https://mlsanity.com/paper/2602.01101/) (2026)
- [From Shallow Humor to Metaphor: Towards Label-Free Harmful Meme Detection via LMM Agent Self-Improvement](https://mlsanity.com/paper/2512.21598/) (2025)
- [All Changes May Have Invariant Principles: Improving Ever-Shifting Harmful Meme Detection via Design Concept Reproduction](https://mlsanity.com/paper/2601.04567/) (2026)
- [Selective Demonstration Retrieval for Improved Implicit Hate Speech Detection](https://mlsanity.com/paper/2504.12082/) (2025)
- [Fall into a Pit, Gain in a Wit: Cognitive-Guided Harmful Meme Detection via Misjudgment Risk Pattern Retrieval](https://mlsanity.com/paper/2510.15946/) (2025)
- [A Multimodal Benchmark for Framing of Oil & Gas Advertising and Potential Greenwashing Detection](https://mlsanity.com/paper/2510.21679/) (2025)
- [MemeArena: Automating Context-Aware Unbiased Evaluation of Harmfulness Understanding for Multimodal Large Language Models](https://mlsanity.com/paper/2510.27196/) (2025)
- [PrismAgent: Illuminating Harm in Memes via a Zero-Shot Interpretable Multi-Agent Framework](https://mlsanity.com/paper/2605.02940/) (2026)
- [FigSIM: A Dataset for Fine-grained Suicide Severity and Figurative Language in Suicide Memes](https://mlsanity.com/paper/2606.02523/) (2026)
- [Beyond a Joke: Multi-Angle Reasoning for Detecting and Explaining Harmful Humor in Memes](https://mlsanity.com/paper/2607.15442/) (2026)
- [SyRHM: Symbolic-Language-Enhanced Reasoning with Associative Retrieval for Zero-shot Harmful Meme Detection](https://mlsanity.com/paper/2609.13794/) (2026)
- [Learn Before You Judge: Progressive Knowledge-to-Decision Alignment for Explainable Hateful Meme Detection](https://mlsanity.com/paper/2609.19778/) (2026)
- [MemeTrace: A Multimodal LLM-Based Agent for Safety-Aware Meme Provenance Analysis](https://mlsanity.com/paper/2610.19267/) (2026)

## Abstract (arXiv)

The proliferation of Internet memes in the age of social media necessitates effective identification of harmful ones. Due to the dynamic nature of memes, existing data-driven models may struggle in low-resource scenarios where only a few labeled examples are available. In this paper, we propose an agency-driven framework for low-resource harmful meme detection, employing both outward and inward analysis with few-shot annotated samples. Inspired by the powerful capacity of Large Multimodal Models (LMMs) on multimodal reasoning, we first retrieve relative memes with annotations to leverage label information as auxiliary signals for the LMM agent. Then, we elicit knowledge-revising behavior within the LMM agent to derive well-generalized insights into meme harmfulness. By combining these strategies, our approach enables dialectical reasoning over intricate and implicit harm-indicative patterns. Extensive experiments conducted on three meme datasets demonstrate that our proposed approach achieves superior performance than state-of-the-art methods on the low-resource harmful meme detection task.