# HumorRank: A Tournament-Based Leaderboard for Evaluating Humor Generation in Large Language Models

arXiv: https://arxiv.org/abs/2604.19786 · HTML: https://mlsanity.com/paper/2604.19786/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 29 of its 54 references that are in the map.
- First posted on arXiv 31 Mar 2026.

## Builds on (most cited first, 29 of 29)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [Qwen3-VL Technical Report](https://mlsanity.com/paper/2511.21631/) (2025)
- [gpt-oss-120b & gpt-oss-20b Model Card](https://mlsanity.com/paper/2508.10925/) (2025)
- [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://mlsanity.com/paper/2403.04132/) (2024)
- [OpenAI GPT-5 System Card](https://mlsanity.com/paper/2601.03267/) (2025)
- [Kimi K2.5: Visual Agentic Intelligence](https://mlsanity.com/paper/2602.02276/) (2026)
- [Open-LLM-Leaderboard: From Multi-choice to Open-style Questions for LLMs Evaluation, Benchmark, and Arena](https://mlsanity.com/paper/2406.07545/) (2024)
- [Let's Think Outside the Box: Exploring Leap-of-Thought in Large Language Models with Creative Humor Generation](https://mlsanity.com/paper/2312.02439/) (2023)
- [Open Ko-LLM Leaderboard: Evaluating Large Language Models in Korean with Ko-H5 Benchmark](https://mlsanity.com/paper/2405.20574/) (2024)
- [Humor in AI: Massive Scale Crowd-Sourced Preferences and Benchmarks for Cartoon Captioning](https://mlsanity.com/paper/2406.10522/) (2024)
- [SemEval-2020 Task 7: Assessing Humor in Edited News Headlines](https://mlsanity.com/paper/2008.00304/) (2020)
- [Innovative Thinking, Infinite Humor: Humor Research of Large Language Models through Structured Thought Leaps](https://mlsanity.com/paper/2410.10370/) (2024)
- [Crowd Score: A Method for the Evaluation of Jokes using Large Language Model AI Voters as Judges](https://mlsanity.com/paper/2212.11214/) (2022)
- [Small But Funny: A Feedback-Driven Approach to Humor Distillation](https://mlsanity.com/paper/2402.18113/) (2024)
- [One Does Not Simply Meme Alone: Evaluating Co-Creativity Between LLMs and Humans in the Generation of Humor](https://mlsanity.com/paper/2501.11433/) (2025)
- [Bridging the Creativity Understanding Gap: Small-Scale Human Alignment Enables Expert-Level Humor Ranking in LLMs](https://mlsanity.com/paper/2502.20356/) (2025)
- [Getting Serious about Humor: Crafting Humor Datasets with Unfunny Large Language Models](https://mlsanity.com/paper/2403.00794/) (2024)
- [Large Language Models for Subjective Language Understanding: A Survey](https://mlsanity.com/paper/2508.07959/) (2025)
- [Which LLMs Get the Joke? Probing Non-STEM Reasoning Abilities with HumorBench](https://mlsanity.com/paper/2507.21476/) (2025)
- [AI Humor Generation: Cognitive, Social and Creative Skills for Effective Humor](https://mlsanity.com/paper/2502.07981/) (2025)
- [Not All Jokes Land: Evaluating Large Language Models Understanding of Workplace Humor](https://mlsanity.com/paper/2506.01819/) (2025)
- [Improving LLM Leaderboards with Psychometrical Methodology](https://mlsanity.com/paper/2501.17200/) (2025)
- [Humor in Pixels: Benchmarking Large Multimodal Models Understanding of Online Comics](https://mlsanity.com/paper/2509.12248/) (2025)
- [Chumor 2.0: Towards Benchmarking Chinese Humor Understanding](https://mlsanity.com/paper/2412.17729/) (2024)
- [From Punchlines to Predictions: A Metric to Assess LLM Performance in Identifying Humor in Stand-Up Comedy](https://mlsanity.com/paper/2504.09049/) (2025)
- [CLARIN-PT-LDB: An Open LLM Leaderboard for Portuguese to assess Language, Culture and Civility](https://mlsanity.com/paper/2603.12872/) (2026)
- [HumorGen: Cognitive Synergy for Humor Generation in Large Language Models via Persona-Based Distillation](https://mlsanity.com/paper/2604.09629/) (2026)

## Abstract (arXiv)

Evaluating humor in large language models (LLMs) is an open challenge because existing approaches yield isolated, incomparable metrics rather than unified model rankings, making it difficult to track progress across systems. We introduce HumorRank, a tournament-based evaluation framework and leaderboard for textual humor generation. On two public benchmarks (SemEval-2026 MWAHAHA and Humor Transfer Bench), we conduct extensive automated pairwise evaluation across nine models spanning proprietary, open-weight, and specialized systems. Pairwise judgments are produced by LLM judges grounded in the General Theory of Verbal Humor (GTVH): each judge integrates structured comedic analysis into adjudication, jointly yielding a preference decision, an interpretable rationale, and mechanism, delivery, and failure tags rather than a black-box funniness score. Judgments are aggregated via an Adaptive Swiss tournament, with Bradley-Terry Maximum Likelihood Estimation (MLE) producing globally consistent humor generation capability rankings. Rankings are cross-judge stable: independent LLM judges (Llama 3.3 70B and Qwen 2.5 72B) yield Kendall tau = 0.889 on both benchmarks, and a human calibration study shows human-LLM agreement tracking human-human agreement on hard funny-versus-funny pairs. Our results demonstrate that HumorRank yields statistically grounded model stratifications, showing that humor quality is associated with mastery of comedic mechanisms such as incongruity, conciseness, escalation, and absurdity rather than model scale alone, with specialized fine-tuned models reaching parity with far larger systems. HumorRank thus provides a scalable, interpretable, and reproducible methodology for benchmarking and understanding LLM-generated humor.