mlsanityData to 7 Oct 2026

MemeArena: Automating Context-Aware Unbiased Evaluation of Harmfulness Understanding for Multimodal Large Language Models

arXiv 2510.27196 ↗First posted 31 Oct 2025cs.CL cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

The proliferation of memes on social media necessitates the capabilities of multimodal Large Language Models (mLLMs) to effectively understand multimodal harmfulness. Existing evaluation approaches predominantly focus on mLLMs' detection accuracy for binary classification tasks, which often fail to reflect the in-depth interpretive nuance of harmfulness across diverse contexts. In this paper, we propose MemeArena, an agent-based arena-style evaluation framework that provides a context-aware and unbiased assessment for mLLMs' understanding of multimodal harmfulness. Specifically, MemeArena simulates diverse interpretive contexts to formulate evaluation tasks that elicit perspective-specific analyses from mLLMs. By integrating varied viewpoints and reaching consensus among evaluators, it enables fair and unbiased comparisons of mLLMs' abilities to interpret multimodal harmfulness. Extensive experiments demonstrate that our framework effectively reduces the evaluation biases of judge agents, with judgment results closely aligning with human preferences, offering valuable insights into reliable and comprehensive mLLM evaluations in multimodal harmfulness understanding. Our code and data are publicly available at https://github.com/Lbotirx/MemeArena.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 34 of its 47 references that are in the map. First posted on arXiv 31 Oct 2025.

Builds on

34 of 47 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025AdamMeme: Adaptively Probe the Reasoning Capacity of Multimodal Large Language Models on Harmfulness2 Jul

Other references · 33

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  3. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  4. 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,906
  5. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
  6. 2024A Survey on LLM-as-a-Judgecited by 1,384
  7. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,348
  8. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  9. 2023Large Language Modelscited by 774
  10. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 651
  11. 2020The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memescited by 485
  12. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 484
21 more
  1. 2024Scaling Synthetic Data Creation with 1,000,000,000 Personascited by 397
  2. 2023Unleashing the Emergent Cognitive Synergy in Large Language Models: A Task-Solving Agent through Multi-Persona Self-Collaborationcited by 246
  3. 2023Is ChatGPT better than Human Annotators? Potential and Limitations of ChatGPT in Explaining Implicit Hate Speechcited by 165
  4. 2019Supervised Multimodal Bitransformers for Classifying Images and Textcited by 130
  5. 2025MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiencycited by 123
  6. 2025Preference Leakage: A Contamination Problem in LLM-as-a-judgecited by 119
  7. 2022Do Androids Laugh at Electric Sheep? Humor "Understanding" Benchmarks from The New Yorker Caption Contestcited by 98
  8. 2025MCP-Universe: Benchmarking Large Language Models with Real-World Model Context Protocol Serverscited by 89
  9. 2021Detecting Harmful Memes and Their Targetscited by 77
  10. 2022Detecting and Understanding Harmful Memes: A Surveycited by 58
  11. 2024GOAT-Bench: Safety Insights to Large Multimodal Models through Meme-Based Social Abusecited by 50
  12. 2023Pro-Cap: Leveraging a Frozen Vision-Language Model for Hateful Meme Detectioncited by 45
  13. 2022Detect Rumors in Microblog Posts for Low-Resource Domains via Adversarial Contrastive Learningcited by 41
  14. 2020Detecting Hate Speech in Multi-modal Memescited by 38
  15. 2023Decoding the Underlying Meaning of Multimodal Hateful Memescited by 38
  16. 2022Zero-Shot Rumor Detection with Propagation Structure via Prompt Learningcited by 32
  17. 2021Rumor Detection on Twitter with Claim-Guided Hierarchical Graph Attention Networkscited by 29
  18. 2024Towards Low-Resource Harmful Meme Detection with LMM Agentscited by 23
  19. 2025FACT-AUDIT: An Adaptive Multi-Agent Framework for Dynamic Fact-Checking Evaluation of Large Language Modelscited by 19
  20. 2024M3Hop-CoT: Misogynous Meme Identification with Multimodal Multi-hop Chain-of-Thoughtcited by 9
  21. 2024Meme Trojan: Backdoor Attacks Against Hateful Meme Detection via Cross-Modal Triggerscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Beneath the Surface: Unveiling Harmful Memes with Multimodal Reasoning Distilled from Large Language Models
  2. 2024Towards Explainable Harmful Meme Detection through Multimodal Debate between Large Language Models
  3. 2025Read as You See: Guiding Unimodal LLMs for Low-Resource Explainable Harmful Meme Detection
  4. 2025MV-Debate: Multi-view Agent Debate with Dynamic Reflection Gating for Multimodal Harmful Content Detection in Social Media
  5. 2025SMARTER: A Data-efficient Framework to Improve Toxicity Detection with Explanation via Self-augmenting Large Language Models
  6. 2025From Recognition to Reasoning: Advancing Multimodal Harmful Meme Detection via Chain-of-Thought Alignment
  7. 2024Just KIDDIN: Knowledge Infusion and Distillation for Detection of INdecent Memes
  8. 2026Beyond a Joke: Multi-Angle Reasoning for Detecting and Explaining Harmful Humor in Memes