# Video Captioning with Aggregated Features Based on Dual Graphs and Gated Fusion

arXiv: https://arxiv.org/abs/2308.06685 · HTML: https://mlsanity.com/paper/2308.06685/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 36 references that are in the map.
- First posted on arXiv 13 Aug 2023.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning](https://mlsanity.com/paper/1602.07261/) (2016)
- [Exploring Visual Relationship for Image Captioning](https://mlsanity.com/paper/1809.07041/) (2018)
- [Sequence to Sequence -- Video to Text](https://mlsanity.com/paper/1505.00487/) (2015)
- [Relation-Aware Graph Attention Network for Visual Question Answering](https://mlsanity.com/paper/1903.12314/) (2019)
- [Object Relational Graph with Teacher-Recommended Learning for Video Captioning](https://mlsanity.com/paper/2002.11566/) (2020)
- [Reconstruction Network for Video Captioning](https://mlsanity.com/paper/1803.11438/) (2018)
- [Translating Videos to Natural Language Using Deep Recurrent Neural Networks](https://mlsanity.com/paper/1412.4729/) (2014)
- [Describing Videos by Exploiting Temporal Structure](https://mlsanity.com/paper/1502.08029/) (2015)
- [Memory-Attended Recurrent Network for Video Captioning](https://mlsanity.com/paper/1905.03966/) (2019)
- [Comprehensive Image Captioning via Scene Graph Decomposition](https://mlsanity.com/paper/2007.11731/) (2020)
- [Controllable Video Captioning with POS Sequence Guidance Based on Gated Fusion Network](https://mlsanity.com/paper/1908.10072/) (2019)
- [Object-aware Aggregation with Bidirectional Temporal Graph for Video Captioning](https://mlsanity.com/paper/1906.04375/) (2019)
- [Semantic Grouping Network for Video Captioning](https://mlsanity.com/paper/2102.00831/) (2021)
- [Learning to Discretely Compose Reasoning Module Networks for Video Captioning](https://mlsanity.com/paper/2007.09049/) (2020)
- [A Semantics-Assisted Video Captioning Model Trained with Scheduled Sampling](https://mlsanity.com/paper/1909.00121/) (2019)
- [Are scene graphs good enough to improve Image Captioning?](https://mlsanity.com/paper/2009.12313/) (2020)
- [Discriminative Latent Semantic Graph for Video Captioning](https://mlsanity.com/paper/2108.03662/) (2021)
- [LiVLR: A Lightweight Visual-Linguistic Reasoning Framework for Video Question Answering](https://mlsanity.com/paper/2111.14547/) (2021)
- [Visual Commonsense-aware Representation Network for Video Captioning](https://mlsanity.com/paper/2211.09469/) (2022)
- [Relation-aware Hierarchical Attention Framework for Video Question Answering](https://mlsanity.com/paper/2105.06160/) (2021)
- [Meaning guided video captioning](https://mlsanity.com/paper/1912.05730/) (2019)
- [Video captioning with stacked attention and semantic hard pull](https://mlsanity.com/paper/2009.07335/) (2020)

## Built on by (most cited first, 1 of 1)

- [Bridging Vision and Language: Modeling Causality and Temporality in Video Narratives](https://mlsanity.com/paper/2412.10720/) (2024)

## Abstract (arXiv)

The application of video captioning models aims at translating the content of videos by using accurate natural language. Due to the complex nature inbetween object interaction in the video, the comprehensive understanding of spatio-temporal relations of objects remains a challenging task. Existing methods often fail in generating sufficient feature representations of video content. In this paper, we propose a video captioning model based on dual graphs and gated fusion: we adapt two types of graphs to generate feature representations of video content and utilize gated fusion to further understand these different levels of information. Using a dual-graphs model to generate appearance features and motion features respectively can utilize the content correlation in frames to generate various features from multiple perspectives. Among them, dual-graphs reasoning can enhance the content correlation in frame sequences to generate advanced semantic features; The gated fusion, on the other hand, aggregates the information in multiple feature representations for comprehensive video content understanding. The experiments conducted on worldly used datasets MSVD and MSR-VTT demonstrate state-of-the-art performance of our proposed approach.