# Chumor 2.0: Towards Benchmarking Chinese Humor Understanding

arXiv: https://arxiv.org/abs/2412.17729 · HTML: https://mlsanity.com/paper/2412.17729/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 51 references that are in the map.
- First posted on arXiv 23 Dec 2024.

## Builds on (most cited first, 22 of 22)

- [Benchmarking Large Language Models for News Summarization](https://mlsanity.com/paper/2301.13848/) (2023)
- [To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning](https://mlsanity.com/paper/2409.12183/) (2024)
- [Evaluating the Logical Reasoning Ability of ChatGPT and GPT-4](https://mlsanity.com/paper/2304.03439/) (2023)
- [Investigating Cultural Alignment of Large Language Models](https://mlsanity.com/paper/2402.13231/) (2024)
- [UR-FUNNY: A Multimodal Language Dataset for Understanding Humor](https://mlsanity.com/paper/1904.06618/) (2019)
- [Survey of Cultural Awareness in Language Models: Text and Beyond](https://mlsanity.com/paper/2411.00860/) (2024)
- [Do Androids Laugh at Electric Sheep? Humor "Understanding" Benchmarks from The New Yorker Caption Contest](https://mlsanity.com/paper/2209.06293/) (2022)
- [LLaMA Beyond English: An Empirical Study on Language Capability Transfer](https://mlsanity.com/paper/2401.01055/) (2024)
- [SemEval-2020 Task 8: Memotion Analysis -- The Visuo-Lingual Metaphor!](https://mlsanity.com/paper/2008.03781/) (2020)
- [All Languages Matter: Evaluating LMMs on Culturally Diverse 100 Languages](https://mlsanity.com/paper/2411.16508/) (2024)
- [Understanding the Capabilities and Limitations of Large Language Models for Cultural Commonsense](https://mlsanity.com/paper/2405.04655/) (2024)
- [COIG-CQIA: Quality is All You Need for Chinese Instruction Fine-tuning](https://mlsanity.com/paper/2403.18058/) (2024)
- [You Are What You Annotate: Towards Better Models through Annotator Representations](https://mlsanity.com/paper/2305.14663/) (2023)
- ["President Vows to Cut <Taxes> Hair": Dataset and Analysis of Creative Text Editing for Humorous Headlines](https://mlsanity.com/paper/1906.00274/) (2019)
- [Chinese Tiny LLM: Pretraining a Chinese-Centric Large Language Model](https://mlsanity.com/paper/2404.04167/) (2024)
- [Task-Adaptive Tokenization: Enhancing Long-Form Text Generation Efficacy in Mental Health and Beyond](https://mlsanity.com/paper/2310.05317/) (2023)
- [Humor in Collective Discourse: Unsupervised Funniness Detection in the New Yorker Cartoon Caption Contest](https://mlsanity.com/paper/1506.08126/) (2015)
- [Bridging Subword Gaps in Pretrain-Finetune Paradigm for Natural Language Generation](https://mlsanity.com/paper/2106.06125/) (2021)
- [Can Pre-trained Language Models Understand Chinese Humor?](https://mlsanity.com/paper/2407.04105/) (2024)
- [Why AI Is WEIRD and Should Not Be This Way: Towards AI For Everyone, With Everyone, By Everyone](https://mlsanity.com/paper/2410.16315/) (2024)
- [ChatGPT for Suicide Risk Assessment on Social Media: Quantitative Evaluation of Model Performance, Potentials and Limitations](https://mlsanity.com/paper/2306.09390/) (2023)
- [Can Language Models Make Fun? A Case Study in Chinese Comical Crosstalk](https://mlsanity.com/paper/2207.00735/) (2022)

## Built on by (most cited first, 4 of 4)

- [GODBench: A Benchmark for Multimodal Large Language Models in Video Comment Art](https://mlsanity.com/paper/2505.11436/) (2025)
- [CFunModel: A "Funny" Language Model Capable of Chinese Humor Generation and Processing](https://mlsanity.com/paper/2503.20417/) (2025)
- [A Survey of Pun Generation: Datasets, Evaluations and Methodologies](https://mlsanity.com/paper/2507.04793/) (2025)
- [HumorRank: A Tournament-Based Leaderboard for Evaluating Humor Generation in Large Language Models](https://mlsanity.com/paper/2604.19786/) (2026)

## Abstract (arXiv)

Existing humor datasets and evaluations predominantly focus on English, leaving limited resources for culturally nuanced humor in non-English languages like Chinese. To address this gap, we construct Chumor, the first Chinese humor explanation dataset that exceeds the size of existing humor datasets. Chumor is sourced from Ruo Zhi Ba, a Chinese Reddit-like platform known for sharing intellectually challenging and culturally specific jokes. We test ten LLMs through direct and chain-of-thought prompting, revealing that Chumor poses significant challenges to existing LLMs, with their accuracy slightly above random and far below human. In addition, our analysis highlights that human-annotated humor explanations are significantly better than those generated by GPT-4o and ERNIE-4-turbo. We release Chumor at https://huggingface.co/datasets/dnaihao/Chumor, our project page is at https://dnaihao.github.io/Chumor-dataset/, our leaderboard is at https://huggingface.co/spaces/dnaihao/Chumor, and our codebase is at https://github.com/dnaihao/Chumor-dataset.