# Video Captioning with Multi-Faceted Attention

arXiv: https://arxiv.org/abs/1612.00234 · HTML: https://mlsanity.com/paper/1612.00234/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Dec 2016.

## Built on by (most cited first, 10 of 10)

- [Retrieving Multimodal Information for Augmented Generation: A Survey](https://mlsanity.com/paper/2303.10868/) (2023)
- [Video-Language Understanding: A Survey from Model Architecture, Model Training, and Data Perspectives](https://mlsanity.com/paper/2406.05615/) (2024)
- [End-to-End Video Question-Answer Generation with Generator-Pretester Network](https://mlsanity.com/paper/2101.01447/) (2021)
- [SEM-POS: Grammatically and Semantically Correct Video Captioning](https://mlsanity.com/paper/2303.14829/) (2023)
- [A Comprehensive Review of the Video-to-Text Problem](https://mlsanity.com/paper/2103.14785/) (2021)
- [Cross-Modal Graph with Meta Concepts for Video Captioning](https://mlsanity.com/paper/2108.06458/) (2021)
- [Video Captioning: a comparative review of where we are and which could be the route](https://mlsanity.com/paper/2204.05976/) (2022)
- [Synchronized Audio-Visual Frames with Fractional Positional Encoding for Transformers in Video-to-Text Translation](https://mlsanity.com/paper/2112.14088/) (2021)
- [EVOQUER: Enhancing Temporal Grounding with Video-Pivoted BackQuery Generation](https://mlsanity.com/paper/2109.04600/) (2021)
- [Exploiting long-term temporal dynamics for video captioning](https://mlsanity.com/paper/2202.10828/) (2022)

## Abstract (arXiv)

Recently, video captioning has been attracting an increasing amount of interest, due to its potential for improving accessibility and information retrieval. While existing methods rely on different kinds of visual features and model structures, they do not fully exploit relevant semantic information. We present an extensible approach to jointly leverage several sorts of visual features and semantic attributes. Our novel architecture builds on LSTMs for sentence generation, with several attention layers and two multimodal layers. The attention mechanism learns to automatically select the most salient visual features or semantic attributes, and the multimodal layer yields overall representations for the input and outputs of the sentence generation component. Experimental results on the challenging MSVD and MSR-VTT datasets show that our framework outperforms the state-of-the-art approaches, while ground truth based semantic attributes are able to further elevate the output quality to a near-human level.