mlsanityData to 14 Oct 2026

SMILE: Multimodal Dataset for Understanding Laughter in Video with Language Models

arXiv 2312.09818 ↗First posted 15 Dec 2023cs.CL cs.AIcited by 28

Not on the map. Why?

Abstract (arXiv)

Despite the recent advances of the artificial intelligence, building social intelligence remains a challenge. Among social signals, laughter is one of the distinctive expressions that occurs during social interactions between humans. In this work, we tackle a new challenge for machines to understand the rationale behind laughter in video, Video Laugh Reasoning. We introduce this new task to explain why people laugh in a particular video and a dataset for this task. Our proposed dataset, SMILE, comprises video clips and language descriptions of why people laugh. We propose a baseline by leveraging the reasoning capacity of large language models (LLMs) with textual video representation. Experiments show that our baseline can generate plausible explanations for laughter. We further investigate the scalability of our baseline by probing other video understanding tasks and in-the-wild videos. We release our dataset, code, and model checkpoints on https://github.com/postech-ami/SMILE-Dataset.

Key facts. Cited by 28 papers in the map (data through 14 Oct 2026). Builds on 37 of its 86 references that are in the map. First posted on arXiv 15 Dec 2023.

Builds on

37 of 86 references.

References · 37

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  6. 2015VQA: Visual Question Answeringcited by 2,506
  7. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  8. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  9. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
  10. 2019How Can We Know What Language Models Know?cited by 959
  11. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 745
  12. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 722
25 more
  1. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 556
  2. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  3. 2018TVQA: Localized, Compositional Video Question Answeringcited by 467
  4. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 447
  5. 2017Tensor Fusion Network for Multimodal Sentiment Analysiscited by 433
  6. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 421
  7. 2015Visual7W: Grounded Question Answering in Imagescited by 318
  8. 2015MovieQA: Understanding Stories in Movies through Question-Answeringcited by 304
  9. 2020MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysiscited by 265
  10. 2019Towards Multimodal Sarcasm Detection (An _Obviously_ Perfect Paper)cited by 174
  11. 2017S3FD: Single Shot Scale-invariant Face Detectorcited by 142
  12. 2019UR-FUNNY: A Multimodal Language Dataset for Understanding Humorcited by 132
  13. 2022Language Models with Image Descriptors are Strong Few-Shot Video-Language Learnerscited by 117
  14. 2021Is Someone Speaking? Exploring Long-term Temporal Features for Audio-visual Active Speaker Detectioncited by 113
  15. 2022Do Androids Laugh at Electric Sheep? Humor "Understanding" Benchmarks from The New Yorker Caption Contestcited by 98
  16. 2015Are You Talking to a Machine? Dataset and Methods for Multilingual Image Question Answeringcited by 80
  17. 2017MovieGraphs: Towards Understanding Human-Centric Situations from Videoscited by 67
  18. 2021Bi-Bimodal Modality Fusion for Correlation-Controlled Multimodal Sentiment Analysiscited by 45
  19. 2015Ask Me Anything: Free-form Visual Question Answering Based on Knowledge from External Sourcescited by 42
  20. 2021Multimodal Learning using Optimal Transport for Sarcasm and Humor Detectioncited by 37
  21. 2022A Multimodal Corpus for Emotion Recognition in Sarcasmcited by 29
  22. 2023LaughTalk: Expressive 3D Talking Head Generation with Laughtercited by 27
  23. 2023CHAMPAGNE: Learning Real-world Conversation from Large-Scale Web Videoscited by 14
  24. 2023Can Language Models Laugh at YouTube Short-form Videos?cited by 11
  25. 2022When to Laugh and How Hard? A Multimodal Approach to Detecting Humor and its Intensitycited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 6; 2025: 11; 2026: 11.

28 papers.

Sort:

By area:

  1. 2024AVHBench: A Cross-Modal Hallucination Benchmark for Audio-Visual Large Language Modelscited by 56
  2. 2025ViSpeak: Visual Instruction Feedback in Streaming Videoscited by 42
  3. 2025EmoBench-M: Benchmarking Emotional Intelligence for Multimodal Large Language Modelscited by 32
  4. 2025video-SALMONN-o1: Reasoning-enhanced Audio-visual Large Language Modelcited by 24
  5. 2024BEAF: Observing BEfore-AFter Changes to Evaluate Hallucination in Vision-language Modelscited by 18
  6. 2025Towards Online Multi-Modal Social Interaction Understandingcited by 12
  7. 2025Multi-speaker Attention Alignment for Multimodal Social Interactioncited by 9
  8. 2026Unveiling the Cognitive Compass: Theory-of-Mind-Guided Multimodal Emotion Reasoningcited by 8
  9. 2025BottleHumor: Self-Informed Humor Explanation using the Information Bottleneck Principlecited by 6
  10. 2024End-to-end Semantic-centric Video-based Multimodal Affective Computingcited by 4
  11. 2025Why We Feel: Breaking Boundaries in Emotional Reasoning with Multimodal Large Language Modelscited by 4
  12. 2025VIBE: Can a VLM Read the Room?cited by 4
16 more
  1. 2026Omni-MMSI: Toward Identity-attributed Social Interaction Understandingcited by 4
  2. 2026SocialDirector: Training-Free Social Interaction Control for Multi-Person Video Generationcited by 4
  3. 2024Selective Vision is the Challenge for Visual Reasoning: A Benchmark for Visual Argument Understandingcited by 3
  4. 2024BQA: Body Language Question Answering Dataset for Video Large Language Modelscited by 3
  5. 2025SatireDecoder: Visual Cascaded Decoupling for Enhancing Satirical Image Comprehensioncited by 3
  6. 2026MTLLFM: Multimodal-Temporal Laughter Localization: UR-FUNNY-Temporal and SMILE-Temporal Benchmarks with an Adaptive Multimodal Fusion Modelcited by 2
  7. 2026SVHalluc: Benchmarking Speech-Vision Hallucination in Audio-Visual Large Language Modelscited by 2
  8. 2025v-HUB: A Benchmark for Video Humor Understanding from Vision and Soundcited by 1
  9. 2026Words at Play: Benchmarking Audio Pun Understanding in Large Audio-Language Modelscited by 1
  10. 2026MMHBench: A Multi-Perspective Benchmark for Mental Health Understanding in Long-Form Videoscited by 1
  11. 2024Generative Emotion Cause Explanation in Multimodal Conversationscited by 0
  12. 2025ManzaiSet: A Multimodal Dataset of Viewer Responses to Japanese Manzai Comedycited by 0
  13. 2026Towards High-Level Semantic Intelligencecited by 0
  14. 2026Reading Between the Frames of Multimodal Drivelologycited by 0
  15. 2026When Does a Laugh Begin? Structured Annotator Disagreement in Temporal Laughter Localizationcited by 0
  16. 2026AI Smart Glasses for Wearable Intelligence: From Egocentric Sensing to Agentic Personalizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026SMILE-Next: Teaching Large Language Models to Detect, Classify, and Reason about Laughter
  2. 2023Laughing Matters: Introducing Laughing-Face Generation using Diffusion Models
  3. 2026Computational Humor with Multimodal LLMs: Methods, Datasets, Evaluation, and Challenges
  4. 2025HUMORCHAIN: Theory-Guided Multi-Stage Reasoning for Interpretable Multimodal Humor Generation
  5. 2024Humor in AI: Massive Scale Crowd-Sourced Preferences and Benchmarks for Cartoon Captioning
  6. 2025SIV-Bench: A Video Benchmark for Social Interaction Understanding and Reasoning
  7. 2025Humor in Pixels: Benchmarking Large Multimodal Models Understanding of Online Comics
  8. 2025Who's Laughing Now? An Overview of Computational Humour Generation and Explanation