mlsanityData to 7 Oct 2026

Do Androids Laugh at Electric Sheep? Humor "Understanding" Benchmarks from The New Yorker Caption Contest

arXiv 2209.06293 ↗First posted 13 Sep 2022cs.CL cs.CVcited by 98

Not on the map. Why?

Cited by 26 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large neural networks can now generate jokes, but do they really "understand" humor? We challenge AI models with three tasks derived from the New Yorker Cartoon Caption Contest: matching a joke to a cartoon, identifying a winning caption, and explaining why a winning caption is funny. These tasks encapsulate progressively more sophisticated aspects of "understanding" a cartoon; key elements are the complex, often surprising relationships between images and captions and the frequent inclusion of indirect and playful allusions to human experience and culture. We investigate both multimodal and language-only models: the former are challenged with the cartoon images directly, while the latter are given multifaceted descriptions of the visual scene to simulate human-level visual understanding. We find that both types of models struggle at all three tasks. For example, our best multimodal models fall 30 accuracy points behind human performance on the matching task, and, even when provided ground-truth visual scene descriptors, human-authored explanations are preferred head-to-head over the best machine-authored ones (few-shot GPT-4) in more than 2/3 of cases. We release models, code, leaderboard, and corpus, which includes newly-gathered annotations describing the image's locations/entities, what's unusual in the scene, and an explanation of the joke.

Key facts. Cited by 98 papers in the map (data through 7 Oct 2026). Builds on 28 of its 94 references that are in the map. First posted on arXiv 13 Sep 2022.

Builds on

28 of 94 references.

References · 28

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  8. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  9. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  10. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  11. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  12. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
16 more
  1. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  2. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  3. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  4. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  5. 2018Hypothesis Only Baselines in Natural Language Inferencecited by 291
  6. 2019Towards Multimodal Sarcasm Detection (An _Obviously_ Perfect Paper)cited by 173
  7. 2019UR-FUNNY: A Multimodal Language Dataset for Understanding Humorcited by 130
  8. 2021Reframing Human-AI Collaboration for Generating Free-Text Explanationscited by 125
  9. 2021Few-Shot Self-Rationalization with Natural Language Promptscited by 83
  10. 2020Does my multimodal model learn cross-modal interactions? It's harder to tell than you might think!cited by 61
  11. 2021On the Diversity and Limits of Human Explanationscited by 30
  12. 2018How Reasonable are Common-Sense Reasoning Tasks: A Case-Study on the Winograd Schema Challenge and SWAGcited by 18
  13. 2015Humor in Collective Discourse: Unsupervised Funniness Detection in the New Yorker Cartoon Caption Contestcited by 16
  14. 2015We Are Humor Beings: Understanding and Predicting Visual Humorcited by 15
  15. 2017Punny Captions: Witty Wordplay in Image Descriptionscited by 4
  16. 2018Neural Joking Machine : Humorous image captioningcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 14; 2024: 36; 2025: 30; 2026: 17.

98 papers.

Sort:

By area:

  1. 2025Gemma 3 Technical Reportcited by 1,686
  2. 2024We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning?cited by 281
  3. 2025BIG-Bench Extra Hardcited by 123
  4. 2023VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Usecited by 100
  5. 2024Language Models Still Struggle to Zero-shot Reason about Time Seriescited by 81
  6. 2023Breaking Common Sense: WHOOPS! A Vision-and-Language Benchmark of Synthetic and Compositional Imagescited by 79
  7. 2023Are You Sure? Challenging LLMs Leads to Performance Drops in The FlipFlop Experimentcited by 71
  8. 2023Let's Think Outside the Box: Exploring Leap-of-Thought in Large Language Models with Creative Humor Generationcited by 56
  9. 2023MemeCap: A Dataset for Captioning and Interpreting Memescited by 46
  10. 2025Neuro-Symbolic AI in 2024: A Systematic Reviewcited by 36
  11. 2023FunQA: Towards Surprising Video Comprehensioncited by 35
  12. 2023MacGyver: Are Large Language Models Creative Problem Solvers?cited by 33
38 more
  1. 2023Probing the Creativity of Large Language Models: Can models produce divergent semantic association?cited by 31
  2. 2023SMILE: Multimodal Dataset for Understanding Laughter in Video with Language Modelscited by 28
  3. 2024Culturally Aware and Adapted NLP: A Taxonomy and a Survey of the State of the Artcited by 26
  4. 2024When LLMs Meet Cunning Texts: A Fallacy Understanding Benchmark for Large Language Modelscited by 22
  5. 2023MMoE: Enhancing Multimodal Models with Mixtures of Multimodal Interaction Expertscited by 21
  6. 2024"A good pun is its own reword": Can Large Language Models Understand Puns?cited by 21
  7. 2024HEMM: Holistic Evaluation of Multimodal Foundation Modelscited by 21
  8. 2023Multimodal Learning Without Labeled Multimodal Data: Guarantees and Applicationscited by 20
  9. 2024Understanding Figurative Meaning through Explainable Visual Entailmentcited by 19
  10. 2024Humor in AI: Massive Scale Crowd-Sourced Preferences and Benchmarks for Cartoon Captioningcited by 18
  11. 2024Enhancing Advanced Visual Reasoning Ability of Large Language Modelscited by 18
  12. 2024Can Large Multimodal Models Uncover Deep Semantics Behind Images?cited by 16
  13. 2024Do Large Language Models Understand Conversational Implicature -- A case study with a chinese sitcomcited by 15
  14. 2024Can MLLMs Understand the Deep Implication Behind Chinese Images?cited by 14
  15. 2024What Are We Measuring When We Evaluate Large Vision-Language Models? An Analysis of Latent Factors and Biasescited by 13
  16. 2022Crowd Score: A Method for the Evaluation of Jokes using Large Language Model AI Voters as Judgescited by 12
  17. 2024Shallow Synthesis of Knowledge in GPT-Generated Texts: A Case Study in Automatic Related Work Compositioncited by 12
  18. 2024VIVA: A Benchmark for Vision-Grounded Decision-Making with Human Valuescited by 12
  19. 2025Bridging the Creativity Understanding Gap: Small-Scale Human Alignment Enables Expert-Level Humor Ranking in LLMscited by 12
  20. 2023Can Language Models Laugh at YouTube Short-form Videos?cited by 11
  21. 2024Cracking the Code of Juxtaposition: Can AI Models Understand the Humorous Contradictionscited by 11
  22. 2024Creating a Lens of Chinese Culture: A Multimodal Dataset for Chinese Pun Rebus Art Understandingcited by 11
  23. 2025The Stochastic Parrot on LLM's Shoulder: A Summative Assessment of Physical Concept Understandingcited by 11
  24. 2024Visual Riddles: a Commonsense and World Knowledge Challenge for Large Vision and Language Modelscited by 10
  25. 2025A Causality-aware Paradigm for Evaluating Creativity of Multimodal Large Language Modelscited by 10
  26. 2023OxfordTVG-HIC: Can Machine Make Humorous Captions from Images?cited by 9
  27. 2024Getting Serious about Humor: Crafting Humor Datasets with Unfunny Large Language Modelscited by 9
  28. 2024YesBut: A High-Quality Annotated Multimodal Dataset for evaluating Satire Comprehension capability of Vision-Language Modelscited by 9
  29. 2024Humor Mechanics: Advancing Humor Generation with Multistep Reasoningcited by 8
  30. 2024Transforming Agency. On the mode of existence of Large Language Modelscited by 7
  31. 2024Creativity in AI: Progresses and Challengescited by 7
  32. 2024HumorDB: Can AI understand graphical humor?cited by 6
  33. 2024Can We Predict Performance of Large Models across Vision-Language Tasks?cited by 6
  34. 2024Self-Comparison for Dataset-Level Membership Inference in Large (Vision-)Language Modelscited by 6
  35. 2025BottleHumor: Self-Informed Humor Explanation using the Information Bottleneck Principlecited by 6
  36. 2025Probing and Inducing Combinational Creativity in Vision-Language Modelscited by 6
  37. 2025Comparing Apples to Oranges: A Dataset & Analysis of LLM Humour Understanding from Traditional Puns to Topical Jokescited by 6
  38. 2025Are Large Pre-trained Vision Language Models Effective Construction Safety Inspectors?cited by 6

50 most cited of 98 All 98 (JSON)