mlsanityData to 14 Oct 2026

Computational Humor with Multimodal LLMs: Methods, Datasets, Evaluation, and Challenges

arXiv 2607.19011 ↗First posted 21 Jul 2026cs.CL cs.AI cs.MMcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Multimodal humor in memes, cartoons, and comics remains difficult for AI systems because intended meaning depends on non-literal mechanisms, shared cultural knowledge, and communicative intent rather than literal scene description. This survey focuses on visual humor understanding in single-image and multi-panel artifacts, while treating humor generation as an emerging downstream frontier. We position the literature against prior humor, sarcasm, and general MLLM surveys and organize it using a capability-centric hierarchy spanning recognition, interpretation and reasoning, and generation. Under this lens, we synthesize benchmark design, evaluation protocols, and modeling paradigms, tracing the field's shift from task-specific fusion models to large-model approaches based on multimodal alignment, evidence-grounded reasoning, and controlled generation. We conclude by highlighting the main barriers to progress: shortcut-prone evaluation, limited cultural and narrative coverage, weak evidence grounding, and unresolved safety and ownership concerns.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 87 of its 121 references that are in the map. First posted on arXiv 21 Jul 2026.

Builds on

87 of 121 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025Humor in Pixels: Benchmarking Large Multimodal Models Understanding of Online Comics12 Sep
  2. 2024MangaUB: A Manga Understanding Benchmark for Large Multimodal Models26 Jul
  3. 2024SoMeLVLM: A Large Vision Language Model for Social Media Processing20 Feb
  4. 2024Can Large Multimodal Models Uncover Deep Semantics Behind Images?17 Feb
  5. 2024Mementos: A Comprehensive Benchmark for Multimodal Large Language Model Reasoning over Image Sequences19 Jan
  6. 2023A Survey on Multimodal Large Language Models23 Jun

Other references · 81

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  4. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
  5. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  6. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 914
  7. 2020The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memescited by 490
  8. 2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainscited by 366
  9. 2022RealTime QA: What's the Answer Right Now?cited by 244
  10. 2025Inference-Time Scaling for Generalist Reward Modelingcited by 239
  11. 2025VisuLogic: A Benchmark for Evaluating Visual Reasoning in Multi-modal Large Language Modelscited by 107
  12. 2024Benchmarking Vision Language Models for Cultural Understandingcited by 101
38 more
  1. 2022Do Androids Laugh at Electric Sheep? Humor "Understanding" Benchmarks from The New Yorker Caption Contestcited by 98
  2. 2021MOMENTA: A Multimodal Framework for Detecting Harmful Memes and Their Targetscited by 86
  3. 2020SemEval-2020 Task 8: Memotion Analysis -- The Visuo-Lingual Metaphor!cited by 78
  4. 2023Language Models, Agent Models, and World Models: The LAW for Machine Reasoning and Planningcited by 60
  5. 2022Detecting and Understanding Harmful Memes: A Surveycited by 58
  6. 2023Let's Think Outside the Box: Exploring Leap-of-Thought in Large Language Models with Creative Humor Generationcited by 56
  7. 2021Disentangling Hate in Online Memescited by 46
  8. 2023MemeCap: A Dataset for Captioning and Interpreting Memescited by 46
  9. 2016Detecting Sarcasm in Multimodal Social Platformscited by 44
  10. 2023Decoding the Underlying Meaning of Multimodal Hateful Memescited by 39
  11. 2024Can Large Language Models Understand Symbolic Graphics Programs?cited by 35
  12. 2024Can Language Models Reason about Individualistic Human Values and Preferences?cited by 34
  13. 2023I Spy a Metaphor: Large Language Models and Diffusion Models Co-Create Visual Metaphorscited by 31
  14. 2024MemeCLIP: Leveraging CLIP Representations for Multimodal Meme Classificationcited by 30
  15. 2022Towards Multi-Modal Sarcasm Detection via Hierarchical Congruity Modeling with Knowledge Enhancementcited by 27
  16. 2025Theory of Mind in Large Language Models: Assessment and Enhancementcited by 25
  17. 2023Improving Hateful Meme Detection through Retrieval-Guided Contrastive Learningcited by 21
  18. 2023MMoE: Enhancing Multimodal Models with Mixtures of Multimodal Interaction Expertscited by 21
  19. 2022DISARM: Detecting the Victims Targeted by Harmful Memescited by 20
  20. 2024MemeGuard: An LLM and VLM-based Framework for Advancing Content Moderation via Meme Interventioncited by 20
  21. 2024Understanding Figurative Meaning through Explainable Visual Entailmentcited by 19
  22. 2024Humor in AI: Massive Scale Crowd-Sourced Preferences and Benchmarks for Cartoon Captioningcited by 19
  23. 2024Evaluating Visual and Cultural Interpretation: The K-Viscuit Benchmark with Human-VLM Collaborationcited by 19
  24. 2024A Survey of Multimodal Sarcasm Detectioncited by 19
  25. 2022What do you MEME? Generating Explanations for Visual Semantic Role Labelling in Memescited by 18
  26. 2015We Are Humor Beings: Understanding and Predicting Visual Humorcited by 15
  27. 2023Beneath the Surface: Unveiling Harmful Memes with Multimodal Reasoning Distilled from Large Language Modelscited by 15
  28. 2020A Multimodal Memes Classification: A Survey and Open Research Issuescited by 14
  29. 2023Characterizing the Entities in Harmful Memes: Who is the Hero, the Villain, the Victim?cited by 14
  30. 2024XMeCap: Meme Caption Generation with Sub-Image Adaptabilitycited by 14
  31. 2024Meme-ingful Analysis: Enhanced Understanding of Cyberbullying in Memes Through Multimodal Explanationscited by 13
  32. 2024Towards Comprehensive Detection of Chinese Harmful Memescited by 13
  33. 2024Innovative Thinking, Infinite Humor: Humor Research of Large Language Models through Structured Thought Leapscited by 13
  34. 2024MemeMQA: Multimodal Question Answering for Memes via Rationale-Based Inferencingcited by 12
  35. 2025Are Vision-Language Models Safe in the Wild? A Meme-Based Benchmark Studycited by 12
  36. 2023MEMEX: Detecting Explanatory Evidence for Memes via Knowledge-Enriched Contextualizationcited by 11
  37. 2024Cracking the Code of Juxtaposition: Can AI Models Understand the Humorous Contradictionscited by 11
  38. 2023OxfordTVG-HIC: Can Machine Make Humorous Captions from Images?cited by 9

50 most cited of 81 All 81 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Culture-Aware Humorous Captioning: Multimodal Humor Generation across Cultural Contexts
  2. 2025From Perception to Punchline: Empowering VLM with the Art of In-the-wild Meme
  3. 2026CaRGo-T: Causal Reasoning Graph-of-Thought improves Multimodal Humor Comprehension
  4. 2025Bridging the Creativity Understanding Gap: Small-Scale Human Alignment Enables Expert-Level Humor Ranking in LLMs
  5. 2025v-HUB: A Benchmark for Video Humor Understanding from Vision and Sound
  6. 2026On the Wings of Imagination: Conflicting Script-based Multi-role Framework for Humor Caption Generation
  7. 2026Learning to Think Like a Cartoon Captionist: Incongruity-Resolution Supervision for Multimodal Humor Understanding
  8. 2026"I See What You Did There": Can Large Vision-Language Models Understand Multimodal Puns?