# UR-FUNNY: A Multimodal Language Dataset for Understanding Humor

arXiv: https://arxiv.org/abs/1904.06618 · HTML: https://mlsanity.com/paper/1904.06618/

## Key facts

- Cited by 132 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Apr 2019.

## Built on by (most cited first, 30 of 132)

- [MultiBench: Multiscale Benchmarks for Multimodal Representation Learning](https://mlsanity.com/paper/2107.07502/) (2021)
- [Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions](https://mlsanity.com/paper/2209.03430/) (2022)
- [Do Androids Laugh at Electric Sheep? Humor "Understanding" Benchmarks from The New Yorker Caption Contest](https://mlsanity.com/paper/2209.06293/) (2022)
- [Quantifying & Modeling Multimodal Interactions: An Information Decomposition Framework](https://mlsanity.com/paper/2302.12247/) (2023)
- [Factorized Contrastive Learning: Going Beyond Multi-view Redundancy](https://mlsanity.com/paper/2306.05268/) (2023)
- [Boosting Multi-modal Model Performance with Adaptive Gradient Modulation](https://mlsanity.com/paper/2308.07686/) (2023)
- [SimMMDG: A Simple and Effective Framework for Multi-modal Domain Generalization](https://mlsanity.com/paper/2310.19795/) (2023)
- [What to align in multimodal contrastive learning?](https://mlsanity.com/paper/2409.07402/) (2024)
- [Bi-Bimodal Modality Fusion for Correlation-Controlled Multimodal Sentiment Analysis](https://mlsanity.com/paper/2107.13669/) (2021)
- [MIntRec: A New Dataset for Multimodal Intent Recognition](https://mlsanity.com/paper/2209.04355/) (2022)
- [Distilling an End-to-End Voice Assistant Without Instruction Training Data](https://mlsanity.com/paper/2410.02678/) (2024)
- [Multimodal Learning using Optimal Transport for Sarcasm and Humor Detection](https://mlsanity.com/paper/2110.10949/) (2021)
- [FunQA: Towards Surprising Video Comprehension](https://mlsanity.com/paper/2306.14899/) (2023)
- [Multi-modal Sarcasm Detection and Humor Classification in Code-mixed Conversations](https://mlsanity.com/paper/2105.09984/) (2021)
- [EmoBench-M: Benchmarking Emotional Intelligence for Multimodal Large Language Models](https://mlsanity.com/paper/2502.04424/) (2025)
- [High-Modality Multimodal Transformer: Quantifying Modality & Interaction Heterogeneity for High-Modality Representation Learning](https://mlsanity.com/paper/2203.01311/) (2022)
- [Understanding the Emergence of Multimodal Representation Alignment](https://mlsanity.com/paper/2502.16282/) (2025)
- [A Low-rank Matching Attention based Cross-modal Feature Fusion Method for Conversational Emotion Recognition](https://mlsanity.com/paper/2306.17799/) (2023)
- [SMILE: Multimodal Dataset for Understanding Laughter in Video with Language Models](https://mlsanity.com/paper/2312.09818/) (2023)
- [i-Code: An Integrative and Composable Multimodal Learning Framework](https://mlsanity.com/paper/2205.01818/) (2022)
- [The MuSe 2022 Multimodal Sentiment Analysis Challenge: Humor, Emotional Reactions, and Stress](https://mlsanity.com/paper/2207.05691/) (2022)
- [The Multimodal Sentiment Analysis in Car Reviews (MuSe-CaR) Dataset: Collection, Insights and Improvements](https://mlsanity.com/paper/2101.06053/) (2021)
- [ExPUNations: Augmenting Puns with Keywords and Explanations](https://mlsanity.com/paper/2210.13513/) (2022)
- [Multimodal Learning Without Labeled Multimodal Data: Guarantees and Applications](https://mlsanity.com/paper/2306.04539/) (2023)
- [MMoE: Enhancing Multimodal Models with Mixtures of Multimodal Interaction Experts](https://mlsanity.com/paper/2311.09580/) (2023)
- [Large Language Models Meet Text-Centric Multimodal Sentiment Analysis: A Survey](https://mlsanity.com/paper/2406.08068/) (2024)
- [Is Sarcasm Detection A Step-by-Step Reasoning Process in Large Language Models?](https://mlsanity.com/paper/2407.12725/) (2024)
- [XMeCap: Meme Caption Generation with Sub-Image Adaptability](https://mlsanity.com/paper/2407.17152/) (2024)
- [Can Pre-trained Language Models Understand Chinese Humor?](https://mlsanity.com/paper/2407.04105/) (2024)
- [The MuSe 2023 Multimodal Sentiment Analysis Challenge: Mimicked Emotions, Cross-Cultural Humour, and Personalisation](https://mlsanity.com/paper/2305.03369/) (2023)

## Abstract (arXiv)

Humor is a unique and creative communicative behavior displayed during social interactions. It is produced in a multimodal manner, through the usage of words (text), gestures (vision) and prosodic cues (acoustic). Understanding humor from these three modalities falls within boundaries of multimodal language; a recent research trend in natural language processing that models natural language as it happens in face-to-face communication. Although humor detection is an established research area in NLP, in a multimodal context it is an understudied area. This paper presents a diverse multimodal dataset, called UR-FUNNY, to open the door to understanding multimodal language used in expressing humor. The dataset and accompanying studies, present a framework in multimodal humor detection for the natural language processing community. UR-FUNNY is publicly available for research.