NarrationBot and InfoBot: A Hybrid System for Automated Video Description
Not on the map. Why?
Abstract (arXiv)
Video accessibility is crucial for blind and low vision users for equitable engagements in education, employment, and entertainment. Despite the availability of professional and amateur services and tools, most human-generated descriptions are expensive and time consuming. Moreover, the rate of human-generated descriptions cannot match the speed of video production. To overcome the increasing gaps in video accessibility, we developed a hybrid system of two tools to 1) automatically generate descriptions for videos and 2) provide answers or additional descriptions in response to user queries on a video. Results from a mixed-methods study with 26 blind and low vision individuals show that our system significantly improved user comprehension and enjoyment of selected videos when both tools were used in tandem. In addition, participants reported no significant difference in their ability to understand videos when presented with autogenerated descriptions versus human-revised autogenerated descriptions. Our results demonstrate user enthusiasm about the developed system and its promise for providing customized access to videos. We discuss the limitations of the current work and provide recommendations for the future development of automated video description tools.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 24 of its 72 references that are in the map. First posted on arXiv 7 Nov 2021.
Builds on
24 of 72 references.
References · 24
- 2017Attention Is All You Needcited by 59,932
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2014Microsoft COCO: Common Objects in Contextcited by 17,498
- 2014Going Deeper with Convolutionscited by 4,231
- 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
- 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
- 2018YOLOv3: An Incremental Improvementcited by 2,652
- 2015VQA: Visual Question Answeringcited by 2,489
- 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
- 2019Towards VQA Models That Can Readcited by 1,696
- 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
12 more
- 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
- 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
- 2017Dense-Captioning Events in Videoscited by 789
- 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
- 2016Visual Dialogcited by 377
- 2015A Dataset for Movie Descriptioncited by 195
- 2019Audio-Visual Scene-Aware Dialogcited by 122
- 2019Iterative Answer Prediction with Pointer-Augmented Multimodal Transformers for TextVQAcited by 107
- 2020TAP: Text-Aware Pre-training for Text-VQA and Text-Captioncited by 95
- 2018Pythia v0.1: the Winning Entry to the VQA Challenge 2018cited by 62
- 2020Spatially Aware Multimodal Transformers for TextVQAcited by 44
- 2020Rescribe: Authoring and Automatically Editing Audio Descriptionscited by 20
Built on by
Citing papers per year: 2024: 1.
1 paper.