# Watch It Twice: Video Captioning with a Refocused Video Encoder

arXiv: https://arxiv.org/abs/1907.12905 · HTML: https://mlsanity.com/paper/1907.12905/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Jul 2019.

## Built on by (most cited first, 9 of 9)

- [A Review of Deep Learning for Video Captioning](https://mlsanity.com/paper/2304.11431/) (2023)
- [GL-RG: Global-Local Representation Granularity for Video Captioning](https://mlsanity.com/paper/2205.10706/) (2022)
- [Movie101v2: Improved Movie Narration Benchmark](https://mlsanity.com/paper/2404.13370/) (2024)
- [End-to-End Video Question-Answer Generation with Generator-Pretester Network](https://mlsanity.com/paper/2101.01447/) (2021)
- [SEAM: Searching Transferable Mixed-Precision Quantization Policy through Large Margin Regularization](https://mlsanity.com/paper/2302.06845/) (2023)
- [Variational Stacked Local Attention Networks for Diverse Video Captioning](https://mlsanity.com/paper/2201.00985/) (2022)
- [Video Captioning: a comparative review of where we are and which could be the route](https://mlsanity.com/paper/2204.05976/) (2022)
- [Edit As You Wish: Video Caption Editing with Multi-grained User Control](https://mlsanity.com/paper/2305.08389/) (2023)
- [Visual-aware Attention Dual-stream Decoder for Video Captioning](https://mlsanity.com/paper/2110.08578/) (2021)

## Abstract (arXiv)

With the rapid growth of video data and the increasing demands of various applications such as intelligent video search and assistance toward visually-impaired people, video captioning task has received a lot of attention recently in computer vision and natural language processing fields. The state-of-the-art video captioning methods focus more on encoding the temporal information, while lack of effective ways to remove irrelevant temporal information and also neglecting the spatial details. However, the current RNN encoding module in single time order can be influenced by the irrelevant temporal information, especially the irrelevant temporal information is at the beginning of the encoding. In addition, neglecting spatial information will lead to the relationship confusion of the words and detailed loss. Therefore, in this paper, we propose a novel recurrent video encoding method and a novel visual spatial feature for the video captioning task. The recurrent encoding module encodes the video twice with the predicted key frame to avoid the irrelevant temporal information often occurring at the beginning and the end of a video. The novel spatial features represent the spatial information in different regions of a video and enrich the details of a caption. Experiments on two benchmark datasets show superior performance of the proposed method.