# Harm or Humor: A Multimodal, Multilingual Benchmark for Overt and Covert Harmful Humor

arXiv: https://arxiv.org/abs/2603.17759 · HTML: https://mlsanity.com/paper/2603.17759/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 53 references that are in the map.
- First posted on arXiv 18 Mar 2026.

## Builds on (most cited first, 21 of 21)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models](https://mlsanity.com/paper/2504.10479/) (2025)
- [MiniCPM-V: A GPT-4V Level MLLM on Your Phone](https://mlsanity.com/paper/2408.01800/) (2024)
- [The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes](https://mlsanity.com/paper/2005.04790/) (2020)
- [VideoChat-Flash: Hierarchical Compression for Long-Context Video Modeling](https://mlsanity.com/paper/2501.00574/) (2024)
- [UR-FUNNY: A Multimodal Language Dataset for Understanding Humor](https://mlsanity.com/paper/1904.06618/) (2019)
- [SemEval-2020 Task 8: Memotion Analysis -- The Visuo-Lingual Metaphor!](https://mlsanity.com/paper/2008.03781/) (2020)
- [Aya Vision: Advancing the Frontier of Multilingual Multimodality](https://mlsanity.com/paper/2505.08751/) (2025)
- ["President Vows to Cut <Taxes> Hair": Dataset and Analysis of Creative Text Editing for Humorous Headlines](https://mlsanity.com/paper/1906.00274/) (2019)
- [Toxic Memes: A Survey of Computational Perspectives on the Detection and Explanation of Meme Toxicities](https://mlsanity.com/paper/2406.07353/) (2024)
- [Getting Serious about Humor: Crafting Humor Datasets with Unfunny Large Language Models](https://mlsanity.com/paper/2403.00794/) (2024)
- [StandUp4AI: A New Multilingual Dataset for Humor Detection in Stand-up Comedy Videos](https://mlsanity.com/paper/2505.18903/) (2025)
- [Text Is Not All You Need: Multimodal Prompting Helps LLMs Understand Humor](https://mlsanity.com/paper/2412.05315/) (2024)
- [HumorDB: Can AI understand graphical humor?](https://mlsanity.com/paper/2406.13564/) (2024)
- [Not All Jokes Land: Evaluating Large Language Models Understanding of Workplace Humor](https://mlsanity.com/paper/2506.01819/) (2025)
- [Comparing Apples to Oranges: A Dataset & Analysis of LLM Humour Understanding from Traditional Puns to Topical Jokes](https://mlsanity.com/paper/2507.13335/) (2025)
- [The MuSe 2024 Multimodal Sentiment Analysis Challenge: Social Perception and Humor Recognition](https://mlsanity.com/paper/2406.07753/) (2024)
- [Deceptive Humor: A Synthetic Multilingual Benchmark Dataset for Bridging Fabricated Claims with Humorous Content](https://mlsanity.com/paper/2503.16031/) (2025)
- [Pun Unintended: LLMs and the Illusion of Humor Understanding](https://mlsanity.com/paper/2509.12158/) (2025)

## Built on by (most cited first, 1 of 1)

- [Refusal is Not Safety! Benchmarking Latent Safety Risks of LLM-Driven Content Humorization](https://mlsanity.com/paper/2607.15977/) (2026)

## Abstract (arXiv)

Dark humor often relies on subtle cultural nuances and implicit cues that require contextual reasoning to interpret, posing safety challenges that current static benchmarks fail to capture. To address this, we introduce a novel multimodal, multilingual benchmark for detecting and understanding harmful and offensive humor. Our manually curated dataset comprises 3,000 texts and 6,000 images in English and Arabic, alongside 1,200 videos that span English, Arabic, and language-independent (universal) contexts. Unlike standard toxicity datasets, we enforce a strict annotation guideline: distinguishing Safe jokes from Harmful ones, with the latter further classified into Explicit (overt) and Implicit (Covert) categories to probe deep reasoning. We systematically evaluate state-of-the-art (SOTA) open and closed-source models across all modalities. Our findings reveal that closed-source models significantly outperform open-source ones, with a notable difference in performance between the English and Arabic languages in both, underscoring the critical need for culturally grounded, reasoning-aware safety alignment. Warning: this paper contains example data that may be offensive, harmful, or biased.