mlsanityData to 7 Oct 2026

Humor in AI: Massive Scale Crowd-Sourced Preferences and Benchmarks for Cartoon Captioning

arXiv 2406.10522 ↗First posted 15 Jun 2024cs.LG cs.AI cs.CLcited by 18

On the map: LLM alignment / RLHF / preference optimisation → Diffusion and multimodal model alignment

Abstract (arXiv)

We present a novel multimodal preference dataset for creative tasks, consisting of over 250 million human ratings on more than 2.2 million captions, collected through crowdsourcing rating data for The New Yorker's weekly cartoon caption contest over the past eight years. This unique dataset supports the development and evaluation of multimodal large language models and preference-based fine-tuning algorithms for humorous caption generation. We propose novel benchmarks for judging the quality of model-generated captions, utilizing both GPT4 and human judgments to establish ranking-based evaluation strategies. Our experimental results highlight the limitations of current fine-tuning methods, such as RLHF and DPO, when applied to creative tasks. Furthermore, we demonstrate that even state-of-the-art models like GPT4 and Claude currently underperform top human contestants in generating humorous captions. As we conclude this extensive data collection effort, we release the entire preference dataset to the research community, fostering further advancements in AI humor generation and evaluation.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Placed in Diffusion and multimodal model alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 52 of its 81 references that are in the map. First posted on arXiv 15 Jun 2024.

Builds on

52 of 81 references.

References · 52

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2017Deep reinforcement learning from human preferencescited by 3,846
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  10. 2023Mistral 7Bcited by 3,065
  11. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  12. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
38 more
  1. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  2. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,068
  3. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  4. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  5. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  6. 2024Self-Rewarding Language Modelscited by 654
  7. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
  8. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  9. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
  10. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  11. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 445
  12. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 399
  13. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
  14. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 386
  15. 2023Statistical Rejection Sampling Improves Preference Optimizationcited by 321
  16. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 288
  17. 2023Preference Ranking Optimization for Human Alignmentcited by 283
  18. 2023Nash Learning from Human Feedbackcited by 220
  19. 2022Fine-tuning language models to find agreement among humans with diverse preferencescited by 189
  20. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferencescited by 176
  21. 2024Generalized Preference Optimization: A Unified Approach to Offline Alignmentcited by 155
  22. 2024A Minimaximalist Approach to Reinforcement Learning from Human Feedbackcited by 141
  23. 2023Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHFcited by 125
  24. 2024MaxMin-RLHF: Alignment with Diverse Human Preferencescited by 121
  25. 2022Do Androids Laugh at Electric Sheep? Humor "Understanding" Benchmarks from The New Yorker Caption Contestcited by 98
  26. 2023Contrastive Preference Learning: Learning from Human Feedback without RLcited by 77
  27. 2024Dense Reward for Free in Reinforcement Learning from Human Feedbackcited by 77
  28. 2023Let's Think Outside the Box: Exploring Leap-of-Thought in Large Language Models with Creative Humor Generationcited by 56
  29. 2023RLCD: Reinforcement Learning from Contrastive Distillation for Language Model Alignmentcited by 42
  30. 2024Dataset Reset Policy Optimization for RLHFcited by 41
  31. 2024Relative Preference Optimization: Enhancing LLM Alignment through Contrasting Responses across Identical and Diverse Promptscited by 36
  32. 2024A Critical Evaluation of AI Feedback for Aligning Large Language Modelscited by 36
  33. 2022Semantic Diversity in Dialogue with Natural Language Inferencecited by 28
  34. 2021Sum of Ranked Range Loss for Supervised Learningcited by 23
  35. 2023ChatGPT is fun, but it is not funny! Humor is still challenging Large Language Modelscited by 23
  36. 2023A density estimation perspective on learning from pairwise human preferencescited by 23
  37. 2024"A good pun is its own reword": Can Large Language Models Understand Puns?cited by 21
  38. 2017A framework for Multi-A(rmed)/B(andit) testing with online FDR controlcited by 17

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 7; 2026: 11.

18 papers.

Sort:

By area:

  1. 2025Gemma 3 Technical Reportcited by 1,686
  2. 2025BIG-Bench Extra Hardcited by 123
  3. 2025Roll the dice & look before you leap: Going beyond the creative limits of next-token predictioncited by 28
  4. 2025Bridging the Creativity Understanding Gap: Small-Scale Human Alignment Enables Expert-Level Humor Ranking in LLMscited by 12
  5. 2025Which LLMs Get the Joke? Probing Non-STEM Reasoning Abilities with HumorBenchcited by 6
  6. 2025Humor in Pixels: Benchmarking Large Multimodal Models Understanding of Online Comicscited by 5
  7. 2026Routing with Generated Data: Annotation-Free LLM Skill Estimation and Expert Selectioncited by 4
  8. 2026On the Wings of Imagination: Conflicting Script-based Multi-role Framework for Humor Caption Generationcited by 3
  9. 2025Balancing Performance and Costs in Best Arm Identificationcited by 1
  10. 2026Cards Against LLMs: Benchmarking Humor Alignment in Large Language Modelscited by 1
  11. 2026Formalizing Latent Thoughts: Four Axioms of Thought Representation in LLMscited by 1
  12. 2026Memes-as-Replies: Can Models Select Humorous Manga Panel Responses?cited by 0
6 more
  1. 2026Learning to Think Like a Cartoon Captionist: Incongruity-Resolution Supervision for Multimodal Humor Understandingcited by 0
  2. 2026Culture-Aware Humorous Captioning: Multimodal Humor Generation across Cultural Contextscited by 0
  3. 2026HumorRank: A Tournament-Based Leaderboard for Evaluating Humor Generation in Large Language Modelscited by 0
  4. 2026Computational Humor with Multimodal LLMs: Methods, Datasets, Evaluation, and Challengescited by 0
  5. 2026Towards High-Level Semantic Intelligencecited by 0
  6. 2026Can MLLMs Decode the Creative Leap? Introducing C4 for Cross-Concept Understandingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026HumorGen: Cognitive Synergy for Humor Generation in Large Language Models via Persona-Based Distillation
  2. 2024Cracking the Code of Juxtaposition: Can AI Models Understand the Humorous Contradictions
  3. 2025HumorPlanSearch: Structured Planning and HuCoT for Contextual AI Humor
  4. 2025From Perception to Punchline: Empowering VLM with the Art of In-the-wild Meme
  5. 2024HumorDB: Can AI understand graphical humor?
  6. 2025HUMORCHAIN: Theory-Guided Multi-Stage Reasoning for Interpretable Multimodal Humor Generation
  7. 2024XMeCap: Meme Caption Generation with Sub-Image Adaptability
  8. 2026Who Laughs with Whom? Disentangling Influential Factors in Humor Preferences across User Clusters and LLMs