mlsanityData to 7 Oct 2026

Cracking the Code of Juxtaposition: Can AI Models Understand the Humorous Contradictions

arXiv 2405.19088 ↗First posted 29 May 2024cs.CL cs.CVcited by 11

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Recent advancements in large multimodal language models have demonstrated remarkable proficiency across a wide range of tasks. Yet, these models still struggle with understanding the nuances of human humor through juxtaposition, particularly when it involves nonlinear narratives that underpin many jokes and humor cues. This paper investigates this challenge by focusing on comics with contradictory narratives, where each comic consists of two panels that create a humorous contradiction. We introduce the YesBut benchmark, which comprises tasks of varying difficulty aimed at assessing AI's capabilities in recognizing and interpreting these comics, ranging from literal content comprehension to deep narrative reasoning. Through extensive experimentation and analysis of recent commercial or open-sourced large (vision) language models, we assess their capability to comprehend the complex interplay of the narrative humor inherent in these comics. Our results show that even state-of-the-art models still lag behind human performance on this task. Our findings offer insights into the current limitations and potential improvements for AI in understanding human creative expressions.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 40 of its 66 references that are in the map. First posted on arXiv 29 May 2024.

Builds on

40 of 66 references.

References · 40

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  6. 2023Mistral 7Bcited by 3,065
  7. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  8. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  9. 2022Emergent Abilities of Large Language Modelscited by 2,483
  10. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  11. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  12. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
28 more
  1. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  2. 2023A Survey on Multimodal Large Language Modelscited by 898
  3. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  4. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  5. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 694
  6. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
  7. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 523
  8. 2024Large Language Models: A Surveycited by 506
  9. 2023PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimizationcited by 311
  10. 2020Evaluation of Text Generation: A Surveycited by 275
  11. 2022Reasoning with Language Model Prompting: A Surveycited by 251
  12. 2023Multimodal Large Language Models: A Surveycited by 193
  13. 2024MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGIcited by 192
  14. 2023VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Usecited by 100
  15. 2022Do Androids Laugh at Electric Sheep? Humor "Understanding" Benchmarks from The New Yorker Caption Contestcited by 98
  16. 2023SEED-Bench-2: Benchmarking Multimodal Large Language Modelscited by 96
  17. 2023Breaking Common Sense: WHOOPS! A Vision-and-Language Benchmark of Synthetic and Compositional Imagescited by 79
  18. 2023Language Models, Agent Models, and World Models: The LAW for Machine Reasoning and Planningcited by 59
  19. 2023MemeCap: A Dataset for Captioning and Interpreting Memescited by 46
  20. 2023CLAIR: Evaluating Image Captions with Large Language Modelscited by 43
  21. 2023ChatGPT is fun, but it is not funny! Humor is still challenging Large Language Modelscited by 23
  22. 2023Eliminating Reasoning via Inferring with Planning: A New Framework to Guide LLMs' Non-linear Thinkingcited by 19
  23. 2015Humor in Collective Discourse: Unsupervised Funniness Detection in the New Yorker Cartoon Caption Contestcited by 16
  24. 2024Can Large Multimodal Models Uncover Deep Semantics Behind Images?cited by 16
  25. 2015We Are Humor Beings: Understanding and Predicting Visual Humorcited by 15
  26. 2023Gemini in Reasoning: Unveiling Commonsense in Multimodal Large Language Modelscited by 15
  27. 2023AMERICANO: Argument Generation with Discourse-driven Decomposition and Agent Interactioncited by 8
  28. 2024Comment-aided Video-Language Alignment via Contrastive Pre-training for Short-form Video Humor Detectioncited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 7; 2026: 4.

11 papers.

  1. 2025Praxis-VLM: Vision-Grounded Decision Making via Text-Driven Reinforcement Learningcited by 9
  2. 2025Large Language Models for Subjective Language Understanding: A Surveycited by 9
  3. 2025Exploring and Evaluating Multimodal Knowledge Reasoning Consistency of Multimodal Large Language Modelscited by 8
  4. 2025BottleHumor: Self-Informed Humor Explanation using the Information Bottleneck Principlecited by 6
  5. 2025Drivel-ology: Challenging LLMs with Interpreting Nonsense with Depthcited by 4
  6. 2025Looking Beyond the Obvious: A Survey on Abstract Concept Recognition for Video Understandingcited by 3
  7. 2026"I See What You Did There": Can Large Vision-Language Models Understand Multimodal Puns?cited by 1
  8. 2025Response Wide Shut? Surprising Observations in Basic Vision Language Model Capabilitiescited by 0
  9. 2026Learning to Think Like a Cartoon Captionist: Incongruity-Resolution Supervision for Multimodal Humor Understandingcited by 0
  10. 2026Computational Humor with Multimodal LLMs: Methods, Datasets, Evaluation, and Challengescited by 0
  11. 2026Towards High-Level Semantic Intelligencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025When 'YES' Meets 'BUT': Can Large Models Comprehend Contradictory Humor Through Comparative Reasoning?
  2. 2025Bridging the Creativity Understanding Gap: Small-Scale Human Alignment Enables Expert-Level Humor Ranking in LLMs
  3. 2025Who's Laughing Now? An Overview of Computational Humour Generation and Explanation
  4. 2025Humor in Pixels: Benchmarking Large Multimodal Models Understanding of Online Comics
  5. 2025Which LLMs Get the Joke? Probing Non-STEM Reasoning Abilities with HumorBench
  6. 2024Humor in AI: Massive Scale Crowd-Sourced Preferences and Benchmarks for Cartoon Captioning
  7. 2025HUMORCHAIN: Theory-Guided Multi-Stage Reasoning for Interpretable Multimodal Humor Generation
  8. 2025Comparing Apples to Oranges: A Dataset & Analysis of LLM Humour Understanding from Traditional Puns to Topical Jokes