# When 'YES' Meets 'BUT': Can Large Models Comprehend Contradictory Humor Through Comparative Reasoning?

arXiv: https://arxiv.org/abs/2503.23137 · HTML: https://mlsanity.com/paper/2503.23137/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 35 of its 79 references that are in the map.
- First posted on arXiv 29 Mar 2025.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)
- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 30 of 35)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [Qwen2 Technical Report](https://mlsanity.com/paper/2407.10671/) (2024)
- [SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension](https://mlsanity.com/paper/2307.16125/) (2023)
- [A Survey on Multimodal Large Language Models](https://mlsanity.com/paper/2306.13549/) (2023)
- [CogVLM: Visual Expert for Pretrained Language Models](https://mlsanity.com/paper/2311.03079/) (2023)
- [Large Language Models: A Survey](https://mlsanity.com/paper/2402.06196/) (2024)
- [PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization](https://mlsanity.com/paper/2306.05087/) (2023)
- [Evaluation of Text Generation: A Survey](https://mlsanity.com/paper/2006.14799/) (2020)
- [CogVLM2: Visual Language Models for Image and Video Understanding](https://mlsanity.com/paper/2408.16500/) (2024)
- [MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI](https://mlsanity.com/paper/2404.16006/) (2024)
- [VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomena](https://mlsanity.com/paper/2112.07566/) (2021)
- [VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Use](https://mlsanity.com/paper/2308.06595/) (2023)
- [SEED-Bench-2: Benchmarking Multimodal Large Language Models](https://mlsanity.com/paper/2311.17092/) (2023)
- [LLM-based NLG Evaluation: Current Status and Challenges](https://mlsanity.com/paper/2402.01383/) (2024)
- [Language Models, Agent Models, and World Models: The LAW for Machine Reasoning and Planning](https://mlsanity.com/paper/2312.05230/) (2023)
- [See, Think, Confirm: Interactive Prompting Between Vision and Language Models for Knowledge-based Visual Reasoning](https://mlsanity.com/paper/2301.05226/) (2023)
- [ChatGPT is fun, but it is not funny! Humor is still challenging Large Language Models](https://mlsanity.com/paper/2306.04563/) (2023)
- [Eliminating Reasoning via Inferring with Planning: A New Framework to Guide LLMs' Non-linear Thinking](https://mlsanity.com/paper/2310.12342/) (2023)
- [Can Large Multimodal Models Uncover Deep Semantics Behind Images?](https://mlsanity.com/paper/2402.11281/) (2024)
- [Humor in Collective Discourse: Unsupervised Funniness Detection in the New Yorker Cartoon Caption Contest](https://mlsanity.com/paper/1506.08126/) (2015)
- [Gemini in Reasoning: Unveiling Commonsense in Multimodal Large Language Models](https://mlsanity.com/paper/2312.17661/) (2023)
- [HumorDB: Can AI understand graphical humor?](https://mlsanity.com/paper/2406.13564/) (2024)

## Built on by (most cited first, 5 of 5)

- [Explain Before You Answer: A Survey on Compositional Visual Reasoning](https://mlsanity.com/paper/2508.17298/) (2025)
- [Which LLMs Get the Joke? Probing Non-STEM Reasoning Abilities with HumorBench](https://mlsanity.com/paper/2507.21476/) (2025)
- [Towards Faithful Reasoning in Comics for Small MLLMs](https://mlsanity.com/paper/2601.02991/) (2026)
- [Computational Humor with Multimodal LLMs: Methods, Datasets, Evaluation, and Challenges](https://mlsanity.com/paper/2607.19011/) (2026)
- [Towards High-Level Semantic Intelligence](https://mlsanity.com/paper/2607.24082/) (2026)

## Abstract (arXiv)

Understanding humor-particularly when it involves complex, contradictory narratives that require comparative reasoning-remains a significant challenge for large vision-language models (VLMs). This limitation hinders AI's ability to engage in human-like reasoning and cultural expression. In this paper, we investigate this challenge through an in-depth analysis of comics that juxtapose panels to create humor through contradictions. We introduce the YesBut (V2), a novel benchmark with 1,262 comic images from diverse multilingual and multicultural contexts, featuring comprehensive annotations that capture various aspects of narrative understanding. Using this benchmark, we systematically evaluate a wide range of VLMs through four complementary tasks spanning from surface content comprehension to deep narrative reasoning, with particular emphasis on comparative reasoning between contradictory elements. Our extensive experiments reveal that even the most advanced models significantly underperform compared to humans, with common failures in visual perception, key element identification, comparative analysis and hallucinations. We further investigate text-based training strategies and social knowledge augmentation methods to enhance model performance. Our findings not only highlight critical weaknesses in VLMs' understanding of cultural and creative expressions but also provide pathways toward developing context-aware models capable of deeper narrative understanding though comparative reasoning.