# A Case Study on Combining ASR and Visual Features for Generating Instructional Video Captions

arXiv: https://arxiv.org/abs/1910.02930 · HTML: https://mlsanity.com/paper/1910.02930/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Oct 2019.

## Built on by (most cited first, 15 of 15)

- [MERLOT: Multimodal Neural Script Knowledge Models](https://mlsanity.com/paper/2106.02636/) (2021)
- [MERLOT Reserve: Neural Script Knowledge through Vision and Language and Sound](https://mlsanity.com/paper/2201.02639/) (2022)
- [OmniVL:One Foundation Model for Image-Language and Video-Language Tasks](https://mlsanity.com/paper/2209.07526/) (2022)
- [End-to-end Generative Pretraining for Multimodal Video Captioning](https://mlsanity.com/paper/2201.08264/) (2022)
- [CrossCLR: Cross-modal Contrastive Learning For Multi-modal Video Representations](https://mlsanity.com/paper/2109.14910/) (2021)
- [Text with Knowledge Graph Augmented Transformer for Video Captioning](https://mlsanity.com/paper/2303.12423/) (2023)
- [MELTR: Meta Loss Transformer for Learning to Fine-tune Video Foundation Models](https://mlsanity.com/paper/2303.13009/) (2023)
- [SoccerNet-Echoes: A Soccer Game Audio Commentary Dataset](https://mlsanity.com/paper/2405.07354/) (2024)
- [CUPID: Adaptive Curation of Pre-training Data for Video-and-Language Representation Learning](https://mlsanity.com/paper/2104.00285/) (2021)
- [What You Say Is What You Show: Visual Narration Detection in Instructional Videos](https://mlsanity.com/paper/2301.02307/) (2023)
- [Deep soccer captioning with transformer: dataset, semantics-related losses, and multi-level evaluation](https://mlsanity.com/paper/2202.05728/) (2022)
- [Multimodal Frame-Scoring Transformer for Video Summarization](https://mlsanity.com/paper/2207.01814/) (2022)
- [Exploring the Role of Audio in Video Captioning](https://mlsanity.com/paper/2306.12559/) (2023)
- [DVCFlow: Modeling Information Flow Towards Human-like Video Captioning](https://mlsanity.com/paper/2111.10146/) (2021)
- [GEM-VPC: A dual Graph-Enhanced Multimodal integration for Video Paragraph Captioning](https://mlsanity.com/paper/2410.09377/) (2024)

## Abstract (arXiv)

Instructional videos get high-traffic on video sharing platforms, and prior work suggests that providing time-stamped, subtask annotations (e.g., "heat the oil in the pan") improves user experiences. However, current automatic annotation methods based on visual features alone perform only slightly better than constant prediction. Taking cues from prior work, we show that we can improve performance significantly by considering automatic speech recognition (ASR) tokens as input. Furthermore, jointly modeling ASR tokens and visual features results in higher performance compared to training individually on either modality. We find that unstated background information is better explained by visual features, whereas fine-grained distinctions (e.g., "add oil" vs. "add olive oil") are disambiguated more easily via ASR tokens.