# Exploration of Visual Features and their weighted-additive fusion for Video Captioning

arXiv: https://arxiv.org/abs/2101.05806 · HTML: https://mlsanity.com/paper/2101.05806/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 32 references that are in the map.
- First posted on arXiv 14 Jan 2021.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning](https://mlsanity.com/paper/1602.07261/) (2016)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [Learning Transferable Architectures for Scalable Image Recognition](https://mlsanity.com/paper/1707.07012/) (2017)
- [Dense-Captioning Events in Videos](https://mlsanity.com/paper/1705.00754/) (2017)
- [Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?](https://mlsanity.com/paper/1711.09577/) (2017)
- [Self-critical Sequence Training for Image Captioning](https://mlsanity.com/paper/1612.00563/) (2016)
- [Meshed-Memory Transformer for Image Captioning](https://mlsanity.com/paper/1912.08226/) (2019)
- [Multi-modal Dense Video Captioning](https://mlsanity.com/paper/2003.07758/) (2020)
- [Reconstruction Network for Video Captioning](https://mlsanity.com/paper/1803.11438/) (2018)
- [A Better Use of Audio-Visual Cues: Dense Video Captioning with Bi-modal Transformer](https://mlsanity.com/paper/2005.08271/) (2020)
- [Memory-Attended Recurrent Network for Video Captioning](https://mlsanity.com/paper/1905.03966/) (2019)
- [Spatio-Temporal Dynamics and Semantic Attribute Enriched Visual Encoding for Video Captioning](https://mlsanity.com/paper/1902.10322/) (2019)
- [Not All Words are Equal: Video-specific Information Loss for Video Captioning](https://mlsanity.com/paper/1901.00097/) (2019)

## Built on by (most cited first, 1 of 1)

- [A Review of Deep Learning for Video Captioning](https://mlsanity.com/paper/2304.11431/) (2023)

## Abstract (arXiv)

Video captioning is a popular task that challenges models to describe events in videos using natural language. In this work, we investigate the ability of various visual feature representations derived from state-of-the-art convolutional neural networks to capture high-level semantic context. We introduce the Weighted Additive Fusion Transformer with Memory Augmented Encoders (WAFTM), a captioning model that incorporates memory in a transformer encoder and uses a novel method, to fuse features, that ensures due importance is given to more significant representations. We illustrate a gain in performance realized by applying Word-Piece Tokenization and a popular REINFORCE algorithm. Finally, we benchmark our model on two datasets and obtain a CIDEr of 92.4 on MSVD and a METEOR of 0.091 on the ActivityNet Captions Dataset.