# Reconstruction Network for Video Captioning

arXiv: https://arxiv.org/abs/1803.11438 · HTML: https://mlsanity.com/paper/1803.11438/

## Key facts

- Cited by 98 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Mar 2018.

## Built on by (most cited first, 30 of 98)

- [Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning](https://mlsanity.com/paper/2302.14115/) (2023)
- [Egocentric Video-Language Pretraining](https://mlsanity.com/paper/2206.01670/) (2022)
- [VLP: A Survey on Vision-Language Pre-training](https://mlsanity.com/paper/2202.09061/) (2022)
- [End-to-End Dense Video Captioning with Parallel Decoding](https://mlsanity.com/paper/2108.07781/) (2021)
- [VidProM: A Million-scale Real Prompt-Gallery Dataset for Text-to-Video Diffusion Models](https://mlsanity.com/paper/2403.06098/) (2024)
- [Video ReCap: Recursive Captioning of Hour-Long Videos](https://mlsanity.com/paper/2402.13250/) (2024)
- [CLIP4Caption: CLIP for Video Caption](https://mlsanity.com/paper/2110.06615/) (2021)
- [VindLU: A Recipe for Effective Video-and-Language Pretraining](https://mlsanity.com/paper/2212.05051/) (2022)
- [VLMbench: A Compositional Benchmark for Vision-and-Language Manipulation](https://mlsanity.com/paper/2206.08522/) (2022)
- [MAFW: A Large-scale, Multi-modal, Compound Affective Database for Dynamic Facial Expression Recognition in the Wild](https://mlsanity.com/paper/2208.00847/) (2022)
- [Towards Surveillance Video-and-Language Understanding: New Dataset, Baselines, and Challenges](https://mlsanity.com/paper/2309.13925/) (2023)
- [VidChapters-7M: Video Chapters at Scale](https://mlsanity.com/paper/2309.13952/) (2023)
- [Procedure-Aware Surgical Video-language Pretraining with Hierarchical Knowledge Augmentation](https://mlsanity.com/paper/2410.00263/) (2024)
- [Do You Remember? Dense Video Captioning with Cross-Modal Memory Retrieval](https://mlsanity.com/paper/2404.07610/) (2024)
- [Hierarchical Modular Network for Video Captioning](https://mlsanity.com/paper/2111.12476/) (2021)
- [Unsupervised Video Object Segmentation via Prototype Memory Network](https://mlsanity.com/paper/2209.03712/) (2022)
- [A Review on Methods and Applications in Multimodal Deep Learning](https://mlsanity.com/paper/2202.09195/) (2022)
- [OphCLIP: Hierarchical Retrieval-Augmented Learning for Ophthalmic Surgical Video-Language Pretraining](https://mlsanity.com/paper/2411.15421/) (2024)
- [A Review of Deep Learning for Video Captioning](https://mlsanity.com/paper/2304.11431/) (2023)
- [Enhancing Self-supervised Video Representation Learning via Multi-level Feature Optimization](https://mlsanity.com/paper/2108.02183/) (2021)
- [Recent Advances and Trends in Multimodal Deep Learning: A Review](https://mlsanity.com/paper/2105.11087/) (2021)
- [Towards Diverse Paragraph Captioning for Untrimmed Videos](https://mlsanity.com/paper/2105.14477/) (2021)
- [Vision+X: A Survey on Multimodal Learning in the Light of Data](https://mlsanity.com/paper/2210.02884/) (2022)
- [Guided Slot Attention for Unsupervised Video Object Segmentation](https://mlsanity.com/paper/2303.08314/) (2023)
- [O2NA: An Object-Oriented Non-Autoregressive Approach for Controllable Video Captioning](https://mlsanity.com/paper/2108.02359/) (2021)
- [Refined Semantic Enhancement towards Frequency Diffusion for Video Captioning](https://mlsanity.com/paper/2211.15076/) (2022)
- [Fine-grained Audible Video Description](https://mlsanity.com/paper/2303.15616/) (2023)
- [GL-RG: Global-Local Representation Granularity for Video Captioning](https://mlsanity.com/paper/2205.10706/) (2022)
- [SMAUG: Sparse Masked Autoencoder for Efficient Video-Language Pre-training](https://mlsanity.com/paper/2211.11446/) (2022)
- [Dual-path Collaborative Generation Network for Emotional Video Captioning](https://mlsanity.com/paper/2408.03006/) (2024)

## Abstract (arXiv)

In this paper, the problem of describing visual contents of a video sequence with natural language is addressed. Unlike previous video captioning work mainly exploiting the cues of video contents to make a language description, we propose a reconstruction network (RecNet) with a novel encoder-decoder-reconstructor architecture, which leverages both the forward (video to sentence) and backward (sentence to video) flows for video captioning. Specifically, the encoder-decoder makes use of the forward flow to produce the sentence description based on the encoded video semantic features. Two types of reconstructors are customized to employ the backward flow and reproduce the video features based on the hidden state sequence generated by the decoder. The generation loss yielded by the encoder-decoder and the reconstruction loss introduced by the reconstructor are jointly drawn into training the proposed RecNet in an end-to-end fashion. Experimental results on benchmark datasets demonstrate that the proposed reconstructor can boost the encoder-decoder models and leads to significant gains in video caption accuracy.