# Dual-path Collaborative Generation Network for Emotional Video Captioning

arXiv: https://arxiv.org/abs/2408.03006 · HTML: https://mlsanity.com/paper/2408.03006/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 168 references that are in the map.
- First posted on arXiv 6 Aug 2024.

## Builds on (most cited first, 30 of 40)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [LightGCN: Simplifying and Powering Graph Convolution Network for Recommendation](https://mlsanity.com/paper/2002.02126/) (2020)
- [VideoBERT: A Joint Model for Video and Language Representation Learning](https://mlsanity.com/paper/1904.01766/) (2019)
- [Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?](https://mlsanity.com/paper/1711.09577/) (2017)
- [GIT: A Generative Image-to-text Transformer for Vision and Language](https://mlsanity.com/paper/2205.14100/) (2022)
- [Gate-Variants of Gated Recurrent Unit (GRU) Neural Networks](https://mlsanity.com/paper/1701.05923/) (2017)
- [A Comprehensive Survey of Deep Learning for Image Captioning](https://mlsanity.com/paper/1810.04020/) (2018)
- [Suppressing Uncertainties for Large-Scale Facial Expression Recognition](https://mlsanity.com/paper/2002.10392/) (2020)
- [Convolutional Networks with Adaptive Inference Graphs](https://mlsanity.com/paper/1711.11503/) (2017)
- [Resolution Adaptive Networks for Efficient Inference](https://mlsanity.com/paper/2003.07326/) (2020)
- [Reconstruction Network for Video Captioning](https://mlsanity.com/paper/1803.11438/) (2018)
- [SCENIC: A JAX Library for Computer Vision Research and Beyond](https://mlsanity.com/paper/2110.11403/) (2021)
- [Less Is More: Picking Informative Frames for Video Captioning](https://mlsanity.com/paper/1803.01457/) (2018)
- [SentiCap: Generating Image Descriptions with Sentiments](https://mlsanity.com/paper/1510.01431/) (2015)
- [Affective Image Content Analysis: Two Decades Review and New Perspectives](https://mlsanity.com/paper/2106.16125/) (2021)
- [DiffNet++: A Neural Influence and Interest Diffusion Network for Social Recommendation](https://mlsanity.com/paper/2002.00844/) (2020)
- [Exploiting multi-CNN features in CNN-RNN based Dimensional Emotion Recognition on the OMG in-the-wild Dataset](https://mlsanity.com/paper/1910.01417/) (2019)
- [From Static to Dynamic: Adapting Landmark-Aware Image Models for Facial Expression Recognition in Videos](https://mlsanity.com/paper/2312.05447/) (2023)
- [Learning Multi-level Deep Representations for Image Emotion Classification](https://mlsanity.com/paper/1611.07145/) (2016)
- [An End-to-End Visual-Audio Attention Network for Emotion Recognition in User-Generated Videos](https://mlsanity.com/paper/2003.00832/) (2020)
- [MDAN: Multi-level Dependent Attention Network for Visual Emotion Analysis](https://mlsanity.com/paper/2203.13443/) (2022)
- [Multi-Attention Network for Compressed Video Referring Object Segmentation](https://mlsanity.com/paper/2207.12622/) (2022)
- [SOLVER: Scene-Object Interrelated Visual Emotion Reasoning Network](https://mlsanity.com/paper/2110.12334/) (2021)
- [Affect2MM: Affective Analysis of Multimedia Content Using Emotion Causality](https://mlsanity.com/paper/2103.06541/) (2021)
- [A Circular-Structured Representation for Visual Emotion Distribution Learning](https://mlsanity.com/paper/2106.12450/) (2021)
- [Improving Image Captioning via Predicting Structured Concepts](https://mlsanity.com/paper/2311.08223/) (2023)
- [CLIP Meets Video Captioning: Concept-Aware Representation Learning Does Matter](https://mlsanity.com/paper/2111.15162/) (2021)

## Built on by (most cited first, 14 of 14)

- [CreatiParser: Generative Image Parsing of Raster Graphic Designs into Editable Layers](https://mlsanity.com/paper/2604.19632/) (2026)
- [IntentVCNet: Bridging Spatio-Temporal Gaps for Intention-Oriented Controllable Video Captioning](https://mlsanity.com/paper/2507.18531/) (2025)
- [EmoVerse: A MLLMs-Driven Emotion Representation Dataset for Interpretable Visual Emotion Analysis](https://mlsanity.com/paper/2511.12554/) (2025)
- [Linguistics-Vision Monotonic Consistent Network for Sign Language Production](https://mlsanity.com/paper/2412.16944/) (2024)
- [Towards Accurate Emotion-Attributed Video Captioning via Fine-grained Emotion-Cause Pair Extraction](https://mlsanity.com/paper/2606.08566/) (2026)
- [Audio-Visual Exchange-Aware Token Pruning for Efficient Audio-Visual Captioning](https://mlsanity.com/paper/2606.10533/) (2026)
- [EmoStyle: Affective Conditioning of Style-Specialist Experts for Emotional Image Generation](https://mlsanity.com/paper/2607.10165/) (2026)
- [Geometry-aware Gaussian Prior and Axial Attention for Cervical Cytology Image Classification](https://mlsanity.com/paper/2607.10278/) (2026)
- [Adaptive Emotional Video Captioning via Affective Heterogeneous Graph Reasoning and Multi-task Joint Learning](https://mlsanity.com/paper/2607.29045/) (2026)
- [Rationale-Guided Knowledge Distillation for Cross-Lingual Stance Detection](https://mlsanity.com/paper/2607.18693/) (2026)
- [DSPO: Diversity-aware Subjective Policy Optimization for Robust Emotional Reasoning](https://mlsanity.com/paper/2609.36775/) (2026)
- [Causal-EVC: Breaking Emotional Spurious Causality via Spatiotemporal Grounding and Counterfactual Intervention](https://mlsanity.com/paper/2609.36776/) (2026)
- [Decoding Affective Nuances: Enhancing MLLMs via Hierarchical Emotion Reasoning and Contrastive Discriminative Pruning](https://mlsanity.com/paper/2609.36782/) (2026)
- [FoveaTree: Information-Density Trees for Query-Guided Visual Token Pruning in Video Large Language Models](https://mlsanity.com/paper/2610.16744/) (2026)

## Abstract (arXiv)

Emotional Video Captioning is an emerging task that aims to describe factual content with the intrinsic emotions expressed in videos. The essential of the EVC task is to effectively perceive subtle and ambiguous visual emotional cues during the caption generation, which is neglected by the traditional video captioning. Existing emotional video captioning methods perceive global visual emotional cues at first, and then combine them with the video features to guide the emotional caption generation, which neglects two characteristics of the EVC task. Firstly, their methods neglect the dynamic subtle changes in the intrinsic emotions of the video, which makes it difficult to meet the needs of common scenes with diverse and changeable emotions. Secondly, as their methods incorporate emotional cues into each step, the guidance role of emotion is overemphasized, which makes factual content more or less ignored during generation. To this end, we propose a dual-path collaborative generation network, which dynamically perceives visual emotional cues evolutions while generating emotional captions by collaborative learning. Specifically, in the dynamic emotion perception path, we propose a dynamic emotion evolution module, which first aggregates visual features and historical caption features to summarize the global visual emotional cues, and then dynamically selects emotional cues required to be re-composed at each stage. Besides, in the adaptive caption generation path, to balance the description of factual content and emotional cues, we propose an emotion adaptive decoder. Thus, our methods can generate emotion-related words at the necessary time step, and our caption generation balances the guidance of factual content and emotional cues well. Extensive experiments on three challenging datasets demonstrate the superiority of our approach and each proposed module.