# Heterogeneous Knowledge Transfer in Video Emotion Recognition, Attribution and Summarization

arXiv: https://arxiv.org/abs/1511.04798 · HTML: https://mlsanity.com/paper/1511.04798/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Nov 2015.

## Built on by (most cited first, 14 of 14)

- [Unlocking the Emotional World of Visual Media: An Overview of the Science, Research, and Impact of Understanding Emotion](https://mlsanity.com/paper/2307.13463/) (2023)
- [StimuVAR: Spatiotemporal Stimuli-aware Video Affective Reasoning with Multimodal Large Language Models](https://mlsanity.com/paper/2409.00304/) (2024)
- [Affective Video Content Analysis: Decade Review and New Perspectives](https://mlsanity.com/paper/2310.17212/) (2023)
- [Weakly-supervised Micro- and Macro-expression Spotting Based on Multi-level Consistency](https://mlsanity.com/paper/2305.02734/) (2023)
- [Use of Affective Visual Information for Summarization of Human-Centric Videos](https://mlsanity.com/paper/2107.03783/) (2021)
- [Taking an Emotional Look at Video Paragraph Captioning](https://mlsanity.com/paper/2203.06356/) (2022)
- [Weak Supervision with Arbitrary Single Frame for Micro- and Macro-expression Spotting](https://mlsanity.com/paper/2403.14240/) (2024)
- [Teaching Human Behavior Improves Content Understanding Abilities Of LLMs](https://mlsanity.com/paper/2405.00942/) (2024)
- [Video Soundtrack Generation by Aligning Emotions and Temporal Boundaries](https://mlsanity.com/paper/2502.10154/) (2025)
- [VEMOCLAP: A video emotion classification web application](https://mlsanity.com/paper/2410.21303/) (2024)
- [A Heterogeneous Multimodal Graph Learning Framework for Recognizing User Emotions in Social Networks](https://mlsanity.com/paper/2501.07746/) (2025)
- [Efficient Video Summarization Framework using EEG and Eye-tracking Signals](https://mlsanity.com/paper/2101.11249/) (2021)
- [PESFormer: Boosting Macro- and Micro-expression Spotting with Direct Timestamp Encoding](https://mlsanity.com/paper/2410.18695/) (2024)
- [Emo-DVS: A Multimodal Benchmark for Privacy-Aware Emotion Recognition with Event Cameras](https://mlsanity.com/paper/2609.06928/) (2026)

## Abstract (arXiv)

Emotion is a key element in user-generated videos. However, it is difficult to understand emotions conveyed in such videos due to the complex and unstructured nature of user-generated content and the sparsity of video frames expressing emotion. In this paper, for the first time, we study the problem of transferring knowledge from heterogeneous external sources, including image and textual data, to facilitate three related tasks in understanding video emotion: emotion recognition, emotion attribution and emotion-oriented summarization. Specifically, our framework (1) learns a video encoding from an auxiliary emotional image dataset in order to improve supervised video emotion recognition, and (2) transfers knowledge from an auxiliary textual corpora for zero-shot recognition of emotion classes unseen during training. The proposed technique for knowledge transfer facilitates novel applications of emotion attribution and emotion-oriented summarization. A comprehensive set of experiments on multiple datasets demonstrate the effectiveness of our framework.