# DVCFlow: Modeling Information Flow Towards Human-like Video Captioning

arXiv: https://arxiv.org/abs/2111.10146 · HTML: https://mlsanity.com/paper/2111.10146/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 48 references that are in the map.
- First posted on arXiv 19 Nov 2021.

## Builds on (most cited first, 30 of 38)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [LXMERT: Learning Cross-Modality Encoder Representations from Transformers](https://mlsanity.com/paper/1908.07490/) (2019)
- [VisualBERT: A Simple and Performant Baseline for Vision and Language](https://mlsanity.com/paper/1908.03557/) (2019)
- [VL-BERT: Pre-training of Generic Visual-Linguistic Representations](https://mlsanity.com/paper/1908.08530/) (2019)
- [Show and Tell: A Neural Image Caption Generator](https://mlsanity.com/paper/1411.4555/) (2014)
- [Dense-Captioning Events in Videos](https://mlsanity.com/paper/1705.00754/) (2017)
- [VideoBERT: A Joint Model for Video and Language Representation Learning](https://mlsanity.com/paper/1904.01766/) (2019)
- [Towards Automatic Learning of Procedures from Web Instructional Videos](https://mlsanity.com/paper/1703.09788/) (2017)
- [Long-term Recurrent Convolutional Networks for Visual Recognition and Description](https://mlsanity.com/paper/1411.4389/) (2014)
- [Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training](https://mlsanity.com/paper/1908.06066/) (2019)
- [ActBERT: Learning Global-Local Video-Text Representations](https://mlsanity.com/paper/2011.07231/) (2020)
- [Scan2Cap: Context-aware Dense Captioning in RGB-D Scans](https://mlsanity.com/paper/2012.02206/) (2020)
- [End-to-End Dense Video Captioning with Masked Transformer](https://mlsanity.com/paper/1804.00819/) (2018)
- [Sequence to Sequence -- Video to Text](https://mlsanity.com/paper/1505.00487/) (2015)
- [Multi-modal Dense Video Captioning](https://mlsanity.com/paper/2003.07758/) (2020)
- [Object Relational Graph with Teacher-Recommended Learning for Video Captioning](https://mlsanity.com/paper/2002.11566/) (2020)
- [Cross-Modal and Hierarchical Modeling of Video and Text](https://mlsanity.com/paper/1810.07212/) (2018)
- [COOT: Cooperative Hierarchical Transformer for Video-Text Representation Learning](https://mlsanity.com/paper/2011.00597/) (2020)
- [Describing Videos by Exploiting Temporal Structure](https://mlsanity.com/paper/1502.08029/) (2015)
- [MART: Memory-Augmented Recurrent Transformer for Coherent Video Paragraph Captioning](https://mlsanity.com/paper/2005.05402/) (2020)
- [Grounded Video Description](https://mlsanity.com/paper/1812.06587/) (2018)
- [A Better Use of Audio-Visual Cues: Dense Video Captioning with Bi-modal Transformer](https://mlsanity.com/paper/2005.08271/) (2020)
- [Bidirectional Attentive Fusion with Context Gating for Dense Video Captioning](https://mlsanity.com/paper/1804.00100/) (2018)
- [Video Paragraph Captioning Using Hierarchical Recurrent Neural Networks](https://mlsanity.com/paper/1510.07712/) (2015)
- [Less Is More: Picking Informative Frames for Video Captioning](https://mlsanity.com/paper/1803.01457/) (2018)
- [Comprehensive Image Captioning via Scene Graph Decomposition](https://mlsanity.com/paper/2007.11731/) (2020)

## Built on by (most cited first, 2 of 2)

- [Dense Video Captioning: A Survey of Techniques, Datasets and Evaluation Protocols](https://mlsanity.com/paper/2311.02538/) (2023)
- [TrafficVLM: A Controllable Visual Language Model for Traffic Video Captioning](https://mlsanity.com/paper/2404.09275/) (2024)

## Abstract (arXiv)

Dense video captioning (DVC) aims to generate multi-sentence descriptions to elucidate the multiple events in the video, which is challenging and demands visual consistency, discoursal coherence, and linguistic diversity. Existing methods mainly generate captions from individual video segments, lacking adaptation to the global visual context and progressive alignment between the fast-evolved visual content and textual descriptions, which results in redundant and spliced descriptions. In this paper, we introduce the concept of information flow to model the progressive information changing across video sequence and captions. By designing a Cross-modal Information Flow Alignment mechanism, the visual and textual information flows are captured and aligned, which endows the captioning process with richer context and dynamics on event/topic evolution. Based on the Cross-modal Information Flow Alignment module, we further put forward DVCFlow framework, which consists of a Global-local Visual Encoder to capture both global features and local features for each video segment, and a pre-trained Caption Generator to produce captions. Extensive experiments on the popular ActivityNet Captions and YouCookII datasets demonstrate that our method significantly outperforms competitive baselines, and generates more human-like text according to subject and objective tests.