# When to Laugh and How Hard? A Multimodal Approach to Detecting Humor and its Intensity

arXiv: https://arxiv.org/abs/2211.01889 · HTML: https://mlsanity.com/paper/2211.01889/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 53 references that are in the map.
- First posted on arXiv 3 Nov 2022.

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [CNN Architectures for Large-Scale Audio Classification](https://mlsanity.com/paper/1609.09430/) (2016)
- [Unsupervised Cross-lingual Representation Learning for Speech Recognition](https://mlsanity.com/paper/2006.13979/) (2020)
- [Towards Multimodal Sarcasm Detection (An _Obviously_ Perfect Paper)](https://mlsanity.com/paper/1906.01815/) (2019)
- [Humor Detection: A Transformer Gets the Last Laugh](https://mlsanity.com/paper/1909.00252/) (2019)
- [DocStruct: A Multimodal Method to Extract Hierarchy Structure in Document for General Form Understanding](https://mlsanity.com/paper/2010.11685/) (2020)
- ["President Vows to Cut <Taxes> Hair": Dataset and Analysis of Creative Text Editing for Humorous Headlines](https://mlsanity.com/paper/1906.00274/) (2019)
- [A Better Variant of Self-Critical Sequence Training](https://mlsanity.com/paper/2003.09971/) (2020)
- [Uncertainty and Surprisal Jointly Deliver the Punchline: Exploiting Incongruity-Based Features for Humor Recognition](https://mlsanity.com/paper/2012.12007/) (2020)
- [!Qu'e maravilla! Multimodal Sarcasm Detection in Spanish: a Dataset and a Baseline](https://mlsanity.com/paper/2105.05542/) (2021)
- [Modelling the Socialization of Creative Agents in a Master-Apprentice Setting: The Case of Movie Title Puns](https://mlsanity.com/paper/1907.04954/) (2019)
- [When a Computer Cracks a Joke: Automated Generation of Humorous Headlines](https://mlsanity.com/paper/2109.08702/) (2021)

## Built on by (most cited first, 5 of 5)

- [SMILE: Multimodal Dataset for Understanding Laughter in Video with Language Models](https://mlsanity.com/paper/2312.09818/) (2023)
- [v-HUB: A Benchmark for Video Humor Understanding from Vision and Sound](https://mlsanity.com/paper/2509.25773/) (2025)
- [Psychology-Driven Enhancement of Humour Translation](https://mlsanity.com/paper/2507.09259/) (2025)
- [When Jokes Cross the Line: Analyzing Regular Humor and Dark Humor in YouTube Shorts](https://mlsanity.com/paper/2606.00046/) (2026)
- [Towards High-Level Semantic Intelligence](https://mlsanity.com/paper/2607.24082/) (2026)

## Abstract (arXiv)

Prerecorded laughter accompanying dialog in comedy TV shows encourages the audience to laugh by clearly marking humorous moments in the show. We present an approach for automatically detecting humor in the Friends TV show using multimodal data. Our model is capable of recognizing whether an utterance is humorous or not and assess the intensity of it. We use the prerecorded laughter in the show as annotation as it marks humor and the length of the audience's laughter tells us how funny a given joke is. We evaluate the model on episodes the model has not been exposed to during the training phase. Our results show that the model is capable of correctly detecting whether an utterance is humorous 78% of the time and how long the audience's laughter reaction should last with a mean absolute error of 600 milliseconds.