# MTLLFM: Multimodal-Temporal Laughter Localization: UR-FUNNY-Temporal and SMILE-Temporal Benchmarks with an Adaptive Multimodal Fusion Model

arXiv: https://arxiv.org/abs/2605.25409 · HTML: https://mlsanity.com/paper/2605.25409/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 40 references that are in the map.
- First posted on arXiv 25 May 2026.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [Qwen2.5-Omni Technical Report](https://mlsanity.com/paper/2503.20215/) (2025)
- [Multimodal Transformer for Unaligned Multimodal Language Sequences](https://mlsanity.com/paper/1906.00295/) (2019)
- [Qwen2-Audio Technical Report](https://mlsanity.com/paper/2407.10759/) (2024)
- [Tensor Fusion Network for Multimodal Sentiment Analysis](https://mlsanity.com/paper/1707.07250/) (2017)
- [MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2005.03545/) (2020)
- [Gated Multimodal Units for Information Fusion](https://mlsanity.com/paper/1702.01992/) (2017)
- [ABAW: Valence-Arousal Estimation, Expression Recognition, Action Unit Detection & Multi-Task Learning Challenges](https://mlsanity.com/paper/2202.10659/) (2022)
- [UR-FUNNY: A Multimodal Language Dataset for Understanding Humor](https://mlsanity.com/paper/1904.06618/) (2019)
- [UntrimmedNets for Weakly Supervised Action Recognition and Detection](https://mlsanity.com/paper/1703.03329/) (2017)
- [LITA: Language Instructed Temporal-Localization Assistant](https://mlsanity.com/paper/2403.19046/) (2024)
- [Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuning](https://mlsanity.com/paper/2406.11161/) (2024)
- [Weakly Supervised Action Localization by Sparse Temporal Pooling Network](https://mlsanity.com/paper/1712.05080/) (2017)
- [Deep Learning-based Action Detection in Untrimmed Videos: A Survey](https://mlsanity.com/paper/2110.00111/) (2021)
- [SMILE: Multimodal Dataset for Understanding Laughter in Video with Language Models](https://mlsanity.com/paper/2312.09818/) (2023)
- [Multimodal Large Language Models Meet Multimodal Emotion Recognition and Reasoning: A Survey](https://mlsanity.com/paper/2509.24322/) (2025)
- [FunnyNet-W: Multimodal Learning of Funny Moments in Videos in the Wild](https://mlsanity.com/paper/2401.04210/) (2024)
- [StandUp4AI: A New Multilingual Dataset for Humor Detection in Stand-up Comedy Videos](https://mlsanity.com/paper/2505.18903/) (2025)
- [D-HUMOR: Dark Humor Understanding via Multimodal Open-ended Reasoning -- A Benchmark Dataset and Method](https://mlsanity.com/paper/2509.06771/) (2025)

## Built on by (most cited first, 2 of 2)

- [TAKE 85: Testing Audiovisual filmmaKer's intEnt across 85 Hours of Film](https://mlsanity.com/paper/2608.30068/) (2026)
- [When Does a Laugh Begin? Structured Annotator Disagreement in Temporal Laughter Localization](https://mlsanity.com/paper/2609.06646/) (2026)

## Abstract (arXiv)

Detecting laughter in video is essential for affective computing and narrative understanding, yet existing approaches treat it as coarse clip-level classification, failing to capture precise temporal boundaries of brief, transient laughter events. We address this gap with two complementary contributions. First, we introduce UR-FUNNY-Temporal and SMILE-Temporal, fully annotated temporal laughter datasets extending two widely-used humor benchmarks. Our annotations cover over 11,053 videos (78.8 hours) and provide precise onset/offset boundaries for each laughter event, along with rich metadata distinguishing speaker vs. audience laughter, modality dominance (acoustic, visual, or both), and intensity levels. Second, we propose a lightweight weakly-supervised framework for temporal laughter localization. Our architecture combines fixed HuBERT and MAE encoders with temporal softmax pooling and adaptive modality gating, learning fine-grained temporal grounding from clip-level labels without requiring frame-level annotations during training. Experiments across three datasets demonstrate that our approach substantially outperforms multimodal foundation models including Gemini 3 Flash, achieving 99% F1 and 68.1% localization precision on sports broadcast data. Ablations validate each architectural component. Furthermore, our precise temporal tags improve downstream laughter reasoning by 227% on CIDEr, enabling GPT-3.5 to outperform GPT-4o. The code, UR-FUNNY-Temporal and SMILE-Temporal datasets are publicly available at https://github.com/WSCSports/MTLLFM-temporal-laughter-localization.