# From Punchlines to Predictions: A Metric to Assess LLM Performance in Identifying Humor in Stand-Up Comedy

arXiv: https://arxiv.org/abs/2504.09049 · HTML: https://mlsanity.com/paper/2504.09049/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 34 references that are in the map.
- First posted on arXiv 12 Apr 2025.

## Builds on (most cited first, 9 of 9)

- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [BARTScore: Evaluating Generated Text as Text Generation](https://mlsanity.com/paper/2106.11520/) (2021)
- [A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT](https://mlsanity.com/paper/2302.11382/) (2023)
- [UR-FUNNY: A Multimodal Language Dataset for Understanding Humor](https://mlsanity.com/paper/1904.06618/) (2019)
- [Are Large Language Model-based Evaluators the Solution to Scaling Up Multilingual Evaluation?](https://mlsanity.com/paper/2309.07462/) (2023)
- [What is the Role of Small Models in the LLM Era: A Survey](https://mlsanity.com/paper/2409.06857/) (2024)
- [Crowd Score: A Method for the Evaluation of Jokes using Large Language Model AI Voters as Judges](https://mlsanity.com/paper/2212.11214/) (2022)
- ["So You Think You're Funny?": Rating the Humour Quotient in Standup Comedy](https://mlsanity.com/paper/2110.12765/) (2021)
- [Text Classification based on Word Subspace with Term-Frequency](https://mlsanity.com/paper/1806.03125/) (2018)

## Built on by (most cited first, 2 of 2)

- [HumorRank: A Tournament-Based Leaderboard for Evaluating Humor Generation in Large Language Models](https://mlsanity.com/paper/2604.19786/) (2026)
- [IROH: Insightful Ranking Of Humor using Multi-Stage Hybrid Retrieval with Rationale-Distilled LLM Judges for JOKER 2026 Track Task 1 English](https://mlsanity.com/paper/2609.15618/) (2026)

## Abstract (arXiv)

Comedy serves as a profound reflection of the times we live in and is a staple element of human interactions. In light of the widespread adoption of Large Language Models (LLMs), the intersection of humor and AI has become no laughing matter. Advancements in the naturalness of human-computer interaction correlates with improvements in AI systems' abilities to understand humor. In this study, we assess the ability of models in accurately identifying humorous quotes from a stand-up comedy transcript. Stand-up comedy's unique comedic narratives make it an ideal dataset to improve the overall naturalness of comedic understanding. We propose a novel humor detection metric designed to evaluate LLMs amongst various prompts on their capability to extract humorous punchlines. The metric has a modular structure that offers three different scoring methods - fuzzy string matching, sentence embedding, and subspace similarity - to provide an overarching assessment of a model's performance. The model's results are compared against those of human evaluators on the same task. Our metric reveals that regardless of prompt engineering, leading models, ChatGPT, Claude, and DeepSeek, achieve scores of at most 51% in humor detection. Notably, this performance surpasses that of humans who achieve a score of 41%. The analysis of human evaluators and LLMs reveals variability in agreement, highlighting the subjectivity inherent in humor and the complexities involved in extracting humorous quotes from live performance transcripts. Code available at https://github.com/swaggirl9000/humor.