# Extracting textual overlays from social media videos using neural networks

arXiv: https://arxiv.org/abs/1804.10687 · HTML: https://mlsanity.com/paper/1804.10687/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Apr 2018.

## Built on by (most cited first, 2 of 2)

- [The Multimodal Sentiment Analysis in Car Reviews (MuSe-CaR) Dataset: Collection, Insights and Improvements](https://mlsanity.com/paper/2101.06053/) (2021)
- [Rooms with Text: A Dataset for Overlaying Text Detection](https://mlsanity.com/paper/2211.11350/) (2022)

## Abstract (arXiv)

Textual overlays are often used in social media videos as people who watch them without the sound would otherwise miss essential information conveyed in the audio stream. This is why extraction of those overlays can serve as an important meta-data source, e.g. for content classification or retrieval tasks. In this work, we present a robust method for extracting textual overlays from videos that builds up on multiple neural network architectures. The proposed solution relies on several processing steps: keyframe extraction, text detection and text recognition. The main component of our system, i.e. the text recognition module, is inspired by a convolutional recurrent neural network architecture and we improve its performance using synthetically generated dataset of over 600,000 images with text prepared by authors specifically for this task. We also develop a filtering method that reduces the amount of overlapping text phrases using Levenshtein distance and further boosts system's performance. The final accuracy of our solution reaches over 80A% and is au pair with state-of-the-art methods.