# Temporal Deformable Convolutional Encoder-Decoder Networks for Video Captioning

arXiv: https://arxiv.org/abs/1905.01077 · HTML: https://mlsanity.com/paper/1905.01077/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 May 2019.

## Built on by (most cited first, 15 of 15)

- [A Keypoint-based Global Association Network for Lane Detection](https://mlsanity.com/paper/2204.07335/) (2022)
- [CoCo-BERT: Improving Video-Language Pre-training with Contrastive Cross-modal Matching and Denoising](https://mlsanity.com/paper/2112.07515/) (2021)
- [Scheduled Sampling in Vision-Language Pretraining with Decoupled Encoder-Decoder Network](https://mlsanity.com/paper/2101.11562/) (2021)
- [Aligning Source Visual and Target Language Domains for Unpaired Video Captioning](https://mlsanity.com/paper/2211.12148/) (2022)
- [Trajectory Prediction with Linguistic Representations](https://mlsanity.com/paper/2110.09741/) (2021)
- [X-modaler: A Versatile and High-performance Codebase for Cross-modal Analytics](https://mlsanity.com/paper/2108.08217/) (2021)
- [Uni-EDEN: Universal Encoder-Decoder Network by Multi-Granular Vision-Language Pre-training](https://mlsanity.com/paper/2201.04026/) (2022)
- [A Comprehensive Review of the Video-to-Text Problem](https://mlsanity.com/paper/2103.14785/) (2021)
- [Non-linear Motion Estimation for Video Frame Interpolation using Space-time Convolutions](https://mlsanity.com/paper/2201.11407/) (2022)
- [Video Captioning: a comparative review of where we are and which could be the route](https://mlsanity.com/paper/2204.05976/) (2022)
- [TLDW: Extreme Multimodal Summarisation of News Videos](https://mlsanity.com/paper/2210.08481/) (2022)
- [Video Captioning in Compressed Video](https://mlsanity.com/paper/2101.00359/) (2021)
- [Visual-aware Attention Dual-stream Decoder for Video Captioning](https://mlsanity.com/paper/2110.08578/) (2021)
- [Text-to-Audio Grounding Based Novel Metric for Evaluating Audio Caption Similarity](https://mlsanity.com/paper/2210.06354/) (2022)
- [Automatic Audio Captioning using Attention weighted Event based Embeddings](https://mlsanity.com/paper/2201.12352/) (2022)

## Abstract (arXiv)

It is well believed that video captioning is a fundamental but challenging task in both computer vision and artificial intelligence fields. The prevalent approach is to map an input video to a variable-length output sentence in a sequence to sequence manner via Recurrent Neural Network (RNN). Nevertheless, the training of RNN still suffers to some degree from vanishing/exploding gradient problem, making the optimization difficult. Moreover, the inherently recurrent dependency in RNN prevents parallelization within a sequence during training and therefore limits the computations. In this paper, we present a novel design --- Temporal Deformable Convolutional Encoder-Decoder Networks (dubbed as TDConvED) that fully employ convolutions in both encoder and decoder networks for video captioning. Technically, we exploit convolutional block structures that compute intermediate states of a fixed number of inputs and stack several blocks to capture long-term relationships. The structure in encoder is further equipped with temporal deformable convolution to enable free-form deformation of temporal sampling. Our model also capitalizes on temporal attention mechanism for sentence generation. Extensive experiments are conducted on both MSVD and MSR-VTT video captioning datasets, and superior results are reported when comparing to conventional RNN-based encoder-decoder techniques. More remarkably, TDConvED increases CIDEr-D performance from 58.8% to 67.2% on MSVD.