# Discriminative Latent Semantic Graph for Video Captioning

arXiv: https://arxiv.org/abs/2108.03662 · HTML: https://mlsanity.com/paper/2108.03662/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 35 references that are in the map.
- First posted on arXiv 8 Aug 2021.

## Builds on (most cited first, 25 of 25)

- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [Improved Training of Wasserstein GANs](https://mlsanity.com/paper/1704.00028/) (2017)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning](https://mlsanity.com/paper/1602.07261/) (2016)
- [Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/1707.07998/) (2017)
- [TEA: Temporal Excitation and Aggregation for Action Recognition](https://mlsanity.com/paper/2004.01398/) (2020)
- [Learning Joint Spatial-Temporal Transformations for Video Inpainting](https://mlsanity.com/paper/2007.10247/) (2020)
- [Sequence to Sequence -- Video to Text](https://mlsanity.com/paper/1505.00487/) (2015)
- [Spatio-Temporal Graph for Video Captioning with Knowledge Distillation](https://mlsanity.com/paper/2003.13942/) (2020)
- [Object Relational Graph with Teacher-Recommended Learning for Video Captioning](https://mlsanity.com/paper/2002.11566/) (2020)
- [Towards a Visual Privacy Advisor: Understanding and Predicting Privacy Risks in Images](https://mlsanity.com/paper/1703.10660/) (2017)
- [Reconstruction Network for Video Captioning](https://mlsanity.com/paper/1803.11438/) (2018)
- [Describing Videos by Exploiting Temporal Structure](https://mlsanity.com/paper/1502.08029/) (2015)
- [Memory-Attended Recurrent Network for Video Captioning](https://mlsanity.com/paper/1905.03966/) (2019)
- [Towards Diverse and Natural Image Descriptions via a Conditional GAN](https://mlsanity.com/paper/1703.06029/) (2017)
- [Less Is More: Picking Informative Frames for Video Captioning](https://mlsanity.com/paper/1803.01457/) (2018)
- [Coherent Multi-Sentence Video Description with Variable Level of Detail](https://mlsanity.com/paper/1403.6173/) (2014)
- [Towards Universal Representation for Unseen Action Recognition](https://mlsanity.com/paper/1803.08460/) (2018)
- [Object-aware Aggregation with Bidirectional Temporal Graph for Video Captioning](https://mlsanity.com/paper/1906.04375/) (2019)
- [Adversarial Inference for Multi-Sentence Video Description](https://mlsanity.com/paper/1812.05634/) (2018)
- [LatentGNN: Learning Efficient Non-local Relations for Visual Recognition](https://mlsanity.com/paper/1905.11634/) (2019)
- [Learning to Discretely Compose Reasoning Module Networks for Video Captioning](https://mlsanity.com/paper/2007.09049/) (2020)
- [Query Twice: Dual Mixture Attention Meta Learning for Video Summarization](https://mlsanity.com/paper/2008.08360/) (2020)

## Built on by (most cited first, 7 of 7)

- [A Review of Deep Learning for Video Captioning](https://mlsanity.com/paper/2304.11431/) (2023)
- [Explainability in Graph Neural Networks: An Experimental Survey](https://mlsanity.com/paper/2203.09258/) (2022)
- [Visual Commonsense-aware Representation Network for Video Captioning](https://mlsanity.com/paper/2211.09469/) (2022)
- [MIR-GAN: Refining Frame-Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognition](https://mlsanity.com/paper/2306.10567/) (2023)
- [Video Captioning: a comparative review of where we are and which could be the route](https://mlsanity.com/paper/2204.05976/) (2022)
- [Pseudo-labeling with Keyword Refining for Few-Supervised Video Captioning](https://mlsanity.com/paper/2411.04059/) (2024)
- [Video Captioning with Aggregated Features Based on Dual Graphs and Gated Fusion](https://mlsanity.com/paper/2308.06685/) (2023)

## Abstract (arXiv)

Video captioning aims to automatically generate natural language sentences that can describe the visual contents of a given video. Existing generative models like encoder-decoder frameworks cannot explicitly explore the object-level interactions and frame-level information from complex spatio-temporal data to generate semantic-rich captions. Our main contribution is to identify three key problems in a joint framework for future video summarization tasks. 1) Enhanced Object Proposal: we propose a novel Conditional Graph that can fuse spatio-temporal information into latent object proposal. 2) Visual Knowledge: Latent Proposal Aggregation is proposed to dynamically extract visual words with higher semantic levels. 3) Sentence Validation: A novel Discriminative Language Validator is proposed to verify generated captions so that key semantic concepts can be effectively preserved. Our experiments on two public datasets (MVSD and MSR-VTT) manifest significant improvements over state-of-the-art approaches on all metrics, especially for BLEU-4 and CIDEr. Our code is available at https://github.com/baiyang4/D-LSG-Video-Caption.