# Syntax Customized Video Captioning by Imitating Exemplar Sentences

arXiv: https://arxiv.org/abs/2112.01062 · HTML: https://mlsanity.com/paper/2112.01062/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 44 references that are in the map.
- First posted on arXiv 2 Dec 2021.

## Builds on (most cited first, 29 of 29)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning](https://mlsanity.com/paper/1602.07261/) (2016)
- [Dense-Captioning Events in Videos](https://mlsanity.com/paper/1705.00754/) (2017)
- [Deep Learning for Text Style Transfer: A Survey](https://mlsanity.com/paper/2011.00416/) (2020)
- [Sequence to Sequence -- Video to Text](https://mlsanity.com/paper/1505.00487/) (2015)
- [Style Transfer in Text: Exploration and Evaluation](https://mlsanity.com/paper/1711.06861/) (2017)
- [From Captions to Visual Concepts and Back](https://mlsanity.com/paper/1411.4952/) (2014)
- [Reconstruction Network for Video Captioning](https://mlsanity.com/paper/1803.11438/) (2018)
- [Translating Videos to Natural Language Using Deep Recurrent Neural Networks](https://mlsanity.com/paper/1412.4729/) (2014)
- [Describing Videos by Exploiting Temporal Structure](https://mlsanity.com/paper/1502.08029/) (2015)
- [Say As You Wish: Fine-grained Control of Image Caption Generation with Abstract Scene Graphs](https://mlsanity.com/paper/2003.00387/) (2020)
- [Unsupervised Image Captioning](https://mlsanity.com/paper/1811.10787/) (2018)
- [Towards Diverse and Natural Image Descriptions via a Conditional GAN](https://mlsanity.com/paper/1703.06029/) (2017)
- [Weakly Supervised Dense Event Captioning in Videos](https://mlsanity.com/paper/1812.03849/) (2018)
- [Fighting Offensive Language on Social Media with Unsupervised Text Style Transfer](https://mlsanity.com/paper/1805.07685/) (2018)
- [Fast, Diverse and Accurate Image Captioning Guided By Part-of-Speech](https://mlsanity.com/paper/1805.12589/) (2018)
- [Jointly Modeling Embedding and Translation to Bridge Video and Language](https://mlsanity.com/paper/1505.01861/) (2015)
- [Controllable Video Captioning with POS Sequence Guidance Based on Gated Fusion Network](https://mlsanity.com/paper/1908.10072/) (2019)
- [Describing like humans: on diversity in image captioning](https://mlsanity.com/paper/1903.12020/) (2019)
- [Coherent Multi-Sentence Video Description with Variable Level of Detail](https://mlsanity.com/paper/1403.6173/) (2014)
- [SemStyle: Learning to Generate Stylised Image Captions using Unaligned Text](https://mlsanity.com/paper/1805.07030/) (2018)
- [Reinforced Video Captioning with Entailment Rewards](https://mlsanity.com/paper/1708.02300/) (2017)
- [Hierarchical Boundary-Aware Neural Encoder for Video Captioning](https://mlsanity.com/paper/1611.09312/) (2016)
- [Contextual Text Style Transfer](https://mlsanity.com/paper/2005.00136/) (2020)
- [Image Captioning at Will: A Versatile Scheme for Effectively Injecting Sentiments into Image Descriptions](https://mlsanity.com/paper/1801.10121/) (2018)
- [Diverse Video Captioning Through Latent Variable Expansion](https://mlsanity.com/paper/1910.12019/) (2019)

## Built on by (most cited first, 2 of 2)

- [Live Video Captioning](https://mlsanity.com/paper/2406.14206/) (2024)
- [METEOR Guided Divergence for Video Captioning](https://mlsanity.com/paper/2212.10690/) (2022)

## Abstract (arXiv)

Enhancing the diversity of sentences to describe video contents is an important problem arising in recent video captioning research. In this paper, we explore this problem from a novel perspective of customizing video captions by imitating exemplar sentence syntaxes. Specifically, given a video and any syntax-valid exemplar sentence, we introduce a new task of Syntax Customized Video Captioning (SCVC) aiming to generate one caption which not only semantically describes the video contents but also syntactically imitates the given exemplar sentence. To tackle the SCVC task, we propose a novel video captioning model, where a hierarchical sentence syntax encoder is firstly designed to extract the syntactic structure of the exemplar sentence, then a syntax conditioned caption decoder is devised to generate the syntactically structured caption expressing video semantics. As there is no available syntax customized groundtruth video captions, we tackle such a challenge by proposing a new training strategy, which leverages the traditional pairwise video captioning data and our collected exemplar sentences to accomplish the model learning. Extensive experiments, in terms of semantic, syntactic, fluency, and diversity evaluations, clearly demonstrate our model capability to generate syntax-varied and semantics-coherent video captions that well imitate different exemplar sentences with enriched diversities.