# AdamMeme: Adaptively Probe the Reasoning Capacity of Multimodal Large Language Models on Harmfulness

arXiv: https://arxiv.org/abs/2507.01702 · HTML: https://mlsanity.com/paper/2507.01702/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 7 of its 37 references that are in the map.
- First posted on arXiv 2 Jul 2025.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)
- Multimodal LLMs / vision-language models / [MLLM training and visual instruction tuning](https://mlsanity.com/mllm/mllm-training-and-visual-instruction-tuning/)

## Builds on (most cited first, 7 of 7)

- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [ChatDev: Communicative Agents for Software Development](https://mlsanity.com/paper/2307.07924/) (2023)
- [Supervised Multimodal Bitransformers for Classifying Images and Text](https://mlsanity.com/paper/1909.02950/) (2019)
- [Detecting Harmful Memes and Their Targets](https://mlsanity.com/paper/2110.00413/) (2021)
- [Decoding the Underlying Meaning of Multimodal Hateful Memes](https://mlsanity.com/paper/2305.17678/) (2023)
- [Detecting Hate Speech in Multi-modal Memes](https://mlsanity.com/paper/2012.14891/) (2020)
- [FACT-AUDIT: An Adaptive Multi-Agent Framework for Dynamic Fact-Checking Evaluation of Large Language Models](https://mlsanity.com/paper/2502.17924/) (2025)

## Built on by (most cited first, 7 of 7)

- [MemeLens: Multilingual Multitask VLMs for Memes](https://mlsanity.com/paper/2601.12539/) (2026)
- [Towards Self-Evolving Benchmarks: Synthesizing Agent Trajectories via Test-Time Exploration under Validate-by-Reproduce Paradigm](https://mlsanity.com/paper/2510.00415/) (2025)
- [MM-CRITIC: A Holistic Evaluation of Large Multimodal Models as Multimodal Critique](https://mlsanity.com/paper/2511.09067/) (2025)
- [All Changes May Have Invariant Principles: Improving Ever-Shifting Harmful Meme Detection via Design Concept Reproduction](https://mlsanity.com/paper/2601.04567/) (2026)
- [MemeArena: Automating Context-Aware Unbiased Evaluation of Harmfulness Understanding for Multimodal Large Language Models](https://mlsanity.com/paper/2510.27196/) (2025)
- [The Safeguard Worked. Is the LLM System Safer?](https://mlsanity.com/paper/2609.00519/) (2026)
- [MetaBench-Harness: Unlocking End-to-End Optimization of Benchmark Harnesses](https://mlsanity.com/paper/2609.33411/) (2026)

## Abstract (arXiv)

The proliferation of multimodal memes in the social media era demands that multimodal Large Language Models (mLLMs) effectively understand meme harmfulness. Existing benchmarks for assessing mLLMs on harmful meme understanding rely on accuracy-based, model-agnostic evaluations using static datasets. These benchmarks are limited in their ability to provide up-to-date and thorough assessments, as online memes evolve dynamically. To address this, we propose AdamMeme, a flexible, agent-based evaluation framework that adaptively probes the reasoning capabilities of mLLMs in deciphering meme harmfulness. Through multi-agent collaboration, AdamMeme provides comprehensive evaluations by iteratively updating the meme data with challenging samples, thereby exposing specific limitations in how mLLMs interpret harmfulness. Extensive experiments show that our framework systematically reveals the varying performance of different target mLLMs, offering in-depth, fine-grained analyses of model-specific weaknesses. Our code is available at https://github.com/Lbotirx/AdamMeme.