# Speech Recognition on TV Series with Video-guided Post-ASR Correction

arXiv: https://arxiv.org/abs/2506.07323 · HTML: https://mlsanity.com/paper/2506.07323/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 24 references that are in the map.
- First posted on arXiv 8 Jun 2025.

## Builds on (most cited first, 18 of 18)

- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [Video-LLaVA: Learning United Visual Representation by Alignment Before Projection](https://mlsanity.com/paper/2311.10122/) (2023)
- [Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models](https://mlsanity.com/paper/2306.05424/) (2023)
- [VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs](https://mlsanity.com/paper/2406.07476/) (2024)
- [Speech Recognition with Deep Recurrent Neural Networks](https://mlsanity.com/paper/1303.5778/) (2013)
- [Libri-Light: A Benchmark for ASR with Limited or No Supervision](https://mlsanity.com/paper/1912.07875/) (2019)
- [Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction](https://mlsanity.com/paper/2201.02184/) (2022)
- [Whispering LLaMA: A Cross-Modal Generative Error Correction Framework for Speech Recognition](https://mlsanity.com/paper/2310.06434/) (2023)
- [VIOLIN: A Large-Scale Dataset for Video-and-Language Inference](https://mlsanity.com/paper/2003.11618/) (2020)
- [AVFormer: Injecting Vision into Frozen Speech Models for Zero-Shot AV-ASR](https://mlsanity.com/paper/2303.16501/) (2023)
- [AVATAR: Unconstrained Audiovisual Speech Recognition](https://mlsanity.com/paper/2206.07684/) (2022)
- [Look, Listen and Recognise: Character-Aware Audio-Visual Subtitling](https://mlsanity.com/paper/2401.12039/) (2024)
- [Generative error correction for code-switching speech recognition using large language models](https://mlsanity.com/paper/2310.13013/) (2023)
- [Fewer Hallucinations, More Verification: A Three-Stage LLM-Based Framework for ASR Error Correction](https://mlsanity.com/paper/2505.24347/) (2025)
- [VILAS: Exploring the Effects of Vision and Language Context in Automatic Speech Recognition](https://mlsanity.com/paper/2305.19972/) (2023)

## Abstract (arXiv)

Automatic Speech Recognition (ASR) has achieved remarkable success with deep learning, driving advancements in conversational artificial intelligence, media transcription, and assistive technologies. However, ASR systems still struggle in complex environments such as TV series, where multiple speakers, overlapping speech, domain-specific terminology, and long-range contextual dependencies pose significant challenges to transcription accuracy. Existing approaches fail to explicitly leverage the rich temporal and contextual information available in the video. To address this limitation, we propose a Video-Guided Post-ASR Correction (VPC) framework that uses a Video-Large Multimodal Model (VLMM) to capture video context and refine ASR outputs. Evaluations on a TV-series benchmark show that our method consistently improves transcription accuracy in complex multimedia environments.