# Neural Language Modeling with Visual Features

arXiv: https://arxiv.org/abs/1903.02930 · HTML: https://mlsanity.com/paper/1903.02930/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Mar 2019.

## Built on by (most cited first, 9 of 9)

- [N24News: A New Dataset for Multimodal News Classification](https://mlsanity.com/paper/2108.13327/) (2021)
- [Multi-modal Machine Learning in Engineering Design: A Review and Future Directions](https://mlsanity.com/paper/2302.10909/) (2023)
- [Multi-modal Machine Learning for Vehicle Rating Predictions Using Image, Text, and Parametric Data](https://mlsanity.com/paper/2305.15218/) (2023)
- [Semantic-Aware Frame-Event Fusion based Pattern Recognition via Large Vision-Language Models](https://mlsanity.com/paper/2311.18592/) (2023)
- [Multimodal Generalized Category Discovery](https://mlsanity.com/paper/2409.11624/) (2024)
- [Towards Better Multi-modal Keyphrase Generation via Visual Entity Enhancement and Multi-granularity Image Noise Filtering](https://mlsanity.com/paper/2309.04734/) (2023)
- [A Transformer-based Prediction Method for Depth of Anesthesia During Target-controlled Infusion of Propofol and Remifentanil](https://mlsanity.com/paper/2308.01929/) (2023)
- [Vision-Assisted Foundation Model for Solving Multi-Task Vehicle Routing Problems](https://mlsanity.com/paper/2606.10431/) (2026)
- [Rethinking the constraints of multimodal fusion: case study in Weakly-Supervised Audio-Visual Video Parsing](https://mlsanity.com/paper/2105.14430/) (2021)

## Abstract (arXiv)

Multimodal language models attempt to incorporate non-linguistic features for the language modeling task. In this work, we extend a standard recurrent neural network (RNN) language model with features derived from videos. We train our models on data that is two orders-of-magnitude bigger than datasets used in prior work. We perform a thorough exploration of model architectures for combining visual and text features. Our experiments on two corpora (YouCookII and 20bn-something-something-v2) show that the best performing architecture consists of middle fusion of visual and text features, yielding over 25% relative improvement in perplexity. We report analysis that provides insights into why our multimodal language model improves upon a standard RNN language model.