mlsanityData to 7 Oct 2026

The Role of the Input in Natural Language Video Description

arXiv 2102.05067 ↗First posted 9 Feb 2021cs.CV cs.CL cs.MMcited by 1

Not on the map. Why?

Abstract (arXiv)

Natural Language Video Description (NLVD) has recently received strong interest in the Computer Vision, Natural Language Processing (NLP), Multimedia, and Autonomous Robotics communities. The State-of-the-Art (SotA) approaches obtained remarkable results when tested on the benchmark datasets. However, those approaches poorly generalize to new datasets. In addition, none of the existing works focus on the processing of the input to the NLVD systems, which is both visual and textual. In this work, it is presented an extensive study dealing with the role of the visual input, evaluated with respect to the overall NLP performance. This is achieved performing data augmentation of the visual component, applying common transformations to model camera distortions, noise, lighting, and camera positioning, that are typical in real-world operative scenarios. A t-SNE based analysis is proposed to evaluate the effects of the considered transformations on the overall visual data distribution. For this study, it is considered the English subset of Microsoft Research Video Description (MSVD) dataset, which is used commonly for NLVD. It was observed that this dataset contains a relevant amount of syntactic and semantic errors. These errors have been amended manually, and the new version of the dataset (called MSVD-v2) is used in the experimentation. The MSVD-v2 dataset is released to help to gain insight into the NLVD problem.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 33 of its 77 references that are in the map. First posted on arXiv 9 Feb 2021.

Builds on

33 of 77 references.

References · 33

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  3. 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,678
  4. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,341
  5. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  6. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
  7. 2016Instance Normalization: The Missing Ingredient for Fast Stylizationcited by 1,187
  8. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  9. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  10. 2018A Joint Sequence Fusion Model for Video Question Answering and Retrievalcited by 208
  11. 2015A Dataset for Movie Descriptioncited by 195
  12. 2016Movie Descriptioncited by 180
21 more
  1. 2015Sequence to Sequence -- Video to Textcited by 157
  2. 2017Data Augmentation for Low-Resource Neural Machine Translationcited by 139
  3. 2018A Closed-form Solution to Photorealistic Image Stylizationcited by 133
  4. 2018Reconstruction Network for Video Captioningcited by 98
  5. 2014Translating Videos to Natural Language Using Deep Recurrent Neural Networkscited by 95
  6. 2017Exploring the structure of a real-time, arbitrary neural artistic stylization networkcited by 89
  7. 2018Style Augmentation: Data Augmentation via Style Randomizationcited by 79
  8. 2017Predicting Visual Features from Text for Image and Video Caption Retrievalcited by 53
  9. 2017Video Captioning via Hierarchical Reinforcement Learningcited by 47
  10. 2016Re-evaluating Automatic Metrics for Image Captioningcited by 45
  11. 2015Using Descriptive Video Services to Create a Large Data Source for Video Annotation Researchcited by 33
  12. 2015Describing Multimedia Content using Attention-based Encoder--Decoder Networkscited by 32
  13. 2015Text Understanding from Scratchcited by 25
  14. 2017Reinforced Video Captioning with Entailment Rewardscited by 21
  15. 2016Hierarchical Boundary-Aware Neural Encoder for Video Captioningcited by 20
  16. 2017Multi-Task Video Captioning with Video and Entailment Generationcited by 17
  17. 2017From Deterministic to Generative: Multi-Modal Stochastic RNNs for Video Captioningcited by 15
  18. 2016Improving LSTM-based Video Description with Linguistic Knowledge Mined from Textcited by 11
  19. 2018End-to-End Video Captioning with Multitask Reinforcement Learningcited by 10
  20. 2017Twitter100k: A Real-world Dataset for Weakly Supervised Cross-Media Retrievalcited by 7
  21. 2016Recognizing and Presenting the Storytelling Video Structure with Deep Multimodal Networkscited by 2

Built on by

Cited by year

Citing papers per year: 2022: 1.

1 paper.

  1. 2022Consistency-based Self-supervised Learning for Temporal Anomaly Localizationcited by 10