# Humor in AI: Massive Scale Crowd-Sourced Preferences and Benchmarks for Cartoon Captioning

arXiv: https://arxiv.org/abs/2406.10522 · HTML: https://mlsanity.com/paper/2406.10522/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion and multimodal model alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 52 of its 81 references that are in the map.
- First posted on arXiv 15 Jun 2024.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Diffusion and multimodal model alignment](https://mlsanity.com/alignment/diffusion-and-multimodal-model-alignment/)

## Builds on (most cited first, 30 of 52)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [Fine-Tuning Language Models from Human Preferences](https://mlsanity.com/paper/1909.08593/) (2019)
- [WebGPT: Browser-assisted question-answering with human feedback](https://mlsanity.com/paper/2112.09332/) (2021)
- [Scaling Laws for Reward Model Overoptimization](https://mlsanity.com/paper/2210.10760/) (2022)
- [A Diversity-Promoting Objective Function for Neural Conversation Models](https://mlsanity.com/paper/1510.03055/) (2015)
- [A General Language Assistant as a Laboratory for Alignment](https://mlsanity.com/paper/2112.00861/) (2021)
- [A General Theoretical Paradigm to Understand Learning from Human Preferences](https://mlsanity.com/paper/2310.12036/) (2023)
- [Self-Rewarding Language Models](https://mlsanity.com/paper/2401.10020/) (2024)
- [RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback](https://mlsanity.com/paper/2309.00267/) (2023)
- [Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models](https://mlsanity.com/paper/2401.01335/) (2024)
- [Reinforced Self-Training (ReST) for Language Modeling](https://mlsanity.com/paper/2308.08998/) (2023)
- [Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision](https://mlsanity.com/paper/2312.09390/) (2023)
- [RRHF: Rank Responses to Align Language Models with Human Feedback without tears](https://mlsanity.com/paper/2304.05302/) (2023)
- [SLiC-HF: Sequence Likelihood Calibration with Human Feedback](https://mlsanity.com/paper/2305.10425/) (2023)
- [Fine-Grained Human Feedback Gives Better Rewards for Language Model Training](https://mlsanity.com/paper/2306.01693/) (2023)
- [Understanding the Effects of RLHF on LLM Generalisation and Diversity](https://mlsanity.com/paper/2310.06452/) (2023)
- [Statistical Rejection Sampling Improves Preference Optimization](https://mlsanity.com/paper/2309.06657/) (2023)
- [Understanding Dataset Difficulty with V-Usable Information](https://mlsanity.com/paper/2110.08420/) (2021)
- [Preference Ranking Optimization for Human Alignment](https://mlsanity.com/paper/2306.17492/) (2023)
- [Nash Learning from Human Feedback](https://mlsanity.com/paper/2312.00886/) (2023)

## Built on by (most cited first, 19 of 19)

- [Gemma 3 Technical Report](https://mlsanity.com/paper/2503.19786/) (2025)
- [BIG-Bench Extra Hard](https://mlsanity.com/paper/2502.19187/) (2025)
- [Roll the dice & look before you leap: Going beyond the creative limits of next-token prediction](https://mlsanity.com/paper/2504.15266/) (2025)
- [Bridging the Creativity Understanding Gap: Small-Scale Human Alignment Enables Expert-Level Humor Ranking in LLMs](https://mlsanity.com/paper/2502.20356/) (2025)
- [Which LLMs Get the Joke? Probing Non-STEM Reasoning Abilities with HumorBench](https://mlsanity.com/paper/2507.21476/) (2025)
- [Humor in Pixels: Benchmarking Large Multimodal Models Understanding of Online Comics](https://mlsanity.com/paper/2509.12248/) (2025)
- [Routing with Generated Data: Annotation-Free LLM Skill Estimation and Expert Selection](https://mlsanity.com/paper/2601.09692/) (2026)
- [On the Wings of Imagination: Conflicting Script-based Multi-role Framework for Humor Caption Generation](https://mlsanity.com/paper/2602.06423/) (2026)
- [Balancing Performance and Costs in Best Arm Identification](https://mlsanity.com/paper/2505.20583/) (2025)
- [Cards Against LLMs: Benchmarking Humor Alignment in Large Language Models](https://mlsanity.com/paper/2604.08757/) (2026)
- [Formalizing Latent Thoughts: Four Axioms of Thought Representation in LLMs](https://mlsanity.com/paper/2606.27378/) (2026)
- [Memes-as-Replies: Can Models Select Humorous Manga Panel Responses?](https://mlsanity.com/paper/2602.15842/) (2026)
- [Learning to Think Like a Cartoon Captionist: Incongruity-Resolution Supervision for Multimodal Humor Understanding](https://mlsanity.com/paper/2604.15210/) (2026)
- [Culture-Aware Humorous Captioning: Multimodal Humor Generation across Cultural Contexts](https://mlsanity.com/paper/2604.18091/) (2026)
- [HumorRank: A Tournament-Based Leaderboard for Evaluating Humor Generation in Large Language Models](https://mlsanity.com/paper/2604.19786/) (2026)
- [Computational Humor with Multimodal LLMs: Methods, Datasets, Evaluation, and Challenges](https://mlsanity.com/paper/2607.19011/) (2026)
- [Towards High-Level Semantic Intelligence](https://mlsanity.com/paper/2607.24082/) (2026)
- [Can MLLMs Decode the Creative Leap? Introducing C4 for Cross-Concept Understanding](https://mlsanity.com/paper/2608.06501/) (2026)
- [Comedic Fool's Gold: Reward Exploits and Countermeasures in Conversational Humor](https://mlsanity.com/paper/2610.00197/) (2026)

## Abstract (arXiv)

We present a novel multimodal preference dataset for creative tasks, consisting of over 250 million human ratings on more than 2.2 million captions, collected through crowdsourcing rating data for The New Yorker's weekly cartoon caption contest over the past eight years. This unique dataset supports the development and evaluation of multimodal large language models and preference-based fine-tuning algorithms for humorous caption generation. We propose novel benchmarks for judging the quality of model-generated captions, utilizing both GPT4 and human judgments to establish ranking-based evaluation strategies. Our experimental results highlight the limitations of current fine-tuning methods, such as RLHF and DPO, when applied to creative tasks. Furthermore, we demonstrate that even state-of-the-art models like GPT4 and Claude currently underperform top human contestants in generating humorous captions. As we conclude this extensive data collection effort, we release the entire preference dataset to the research community, fostering further advancements in AI humor generation and evaluation.