# Event and Entity Extraction from Generated Video Captions

arXiv: https://arxiv.org/abs/2211.02982 · HTML: https://mlsanity.com/paper/2211.02982/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 46 references that are in the map.
- First posted on arXiv 5 Nov 2022.

## Builds on (most cited first, 18 of 18)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Convolutional Neural Networks for Sentence Classification](https://mlsanity.com/paper/1408.5882/) (2014)
- [Temporal Segment Networks: Towards Good Practices for Deep Action Recognition](https://mlsanity.com/paper/1608.00859/) (2016)
- [Dense-Captioning Events in Videos](https://mlsanity.com/paper/1705.00754/) (2017)
- [Neural Architectures for Named Entity Recognition](https://mlsanity.com/paper/1603.01360/) (2016)
- [Universal Dependencies v2: An Evergrowing Multilingual Treebank Collection](https://mlsanity.com/paper/2004.10643/) (2020)
- [End-to-End Dense Video Captioning with Masked Transformer](https://mlsanity.com/paper/1804.00819/) (2018)
- [End-to-End Dense Video Captioning with Parallel Decoding](https://mlsanity.com/paper/2108.07781/) (2021)
- [Grounded Video Description](https://mlsanity.com/paper/1812.06587/) (2018)
- [MART: Memory-Augmented Recurrent Transformer for Coherent Video Paragraph Captioning](https://mlsanity.com/paper/2005.05402/) (2020)
- [A Better Use of Audio-Visual Cues: Dense Video Captioning with Bi-modal Transformer](https://mlsanity.com/paper/2005.08271/) (2020)
- [Deep Reinforcement Learning for Mention-Ranking Coreference Models](https://mlsanity.com/paper/1609.08667/) (2016)
- [Jointly Localizing and Describing Events for Dense Video Captioning](https://mlsanity.com/paper/1804.08274/) (2018)
- [Video Paragraph Captioning Using Hierarchical Recurrent Neural Networks](https://mlsanity.com/paper/1510.07712/) (2015)
- [A Survey on Open Information Extraction](https://mlsanity.com/paper/1806.05599/) (2018)
- [Attention Clusters: Purely Attention Based Local Feature Integration for Video Classification](https://mlsanity.com/paper/1711.09550/) (2017)
- [Exploring Emotion Features and Fusion Strategies for Audio-Video Emotion Recognition](https://mlsanity.com/paper/2012.13912/) (2020)
- [Bag-of-Words vs. Graph vs. Sequence in Text Classification: Questioning the Necessity of Text-Graphs and the Surprising Strength of a Wide MLP](https://mlsanity.com/paper/2109.03777/) (2021)

## Abstract (arXiv)

Annotation of multimedia data by humans is time-consuming and costly, while reliable automatic generation of semantic metadata is a major challenge. We propose a framework to extract semantic metadata from automatically generated video captions. As metadata, we consider entities, the entities' properties, relations between entities, and the video category. We employ two state-of-the-art dense video captioning models with masked transformer (MT) and parallel decoding (PVDC) to generate captions for videos of the ActivityNet Captions dataset. Our experiments show that it is possible to extract entities, their properties, relations between entities, and the video category from the generated captions. We observe that the quality of the extracted information is mainly influenced by the quality of the event localization in the video as well as the performance of the event caption generation.