# Refining Transcripts With TV Subtitles by Prompt-Based Weakly Supervised Training of ASR

arXiv: https://arxiv.org/abs/2509.04491 · HTML: https://mlsanity.com/paper/2509.04491/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 20 references that are in the map.
- First posted on arXiv 1 Sep 2025.

## Builds on (most cited first, 11 of 11)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [A Comprehensive Survey of Few-shot Learning: Evolution, Applications, Challenges, and Opportunities](https://mlsanity.com/paper/2205.06743/) (2022)
- [Summary of ChatGPT-Related Research and Perspective Towards the Future of Large Language Models](https://mlsanity.com/paper/2304.01852/) (2023)
- [Self-Training: A Survey](https://mlsanity.com/paper/2202.12040/) (2022)
- [Careless Whisper: Speech-to-Text Hallucination Harms](https://mlsanity.com/paper/2402.08021/) (2024)
- [Prompting the Hidden Talent of Web-Scale Speech Models for Zero-Shot Task Generalization](https://mlsanity.com/paper/2305.11095/) (2023)
- [Comparison of Multilingual Self-Supervised and Weakly-Supervised Speech Pre-Training for Adaptation to Unseen Languages](https://mlsanity.com/paper/2305.12606/) (2023)
- [Extending Whisper with prompt tuning to target-speaker ASR](https://mlsanity.com/paper/2312.08079/) (2023)
- [Zero-shot Domain-sensitive Speech Recognition with Prompt-conditioning Fine-tuning](https://mlsanity.com/paper/2307.10274/) (2023)
- [Large scale weakly and semi-supervised learning for low-resource video ASR](https://mlsanity.com/paper/2005.07850/) (2020)
- [Prompting Whisper for QA-driven Zero-shot End-to-end Spoken Language Understanding](https://mlsanity.com/paper/2406.15209/) (2024)

## Abstract (arXiv)

This study proposes a novel approach to using TV subtitles within a weakly supervised (WS) Automatic Speech Recognition (ASR) framework. Although TV subtitles are readily available, their imprecise alignment with corresponding audio limits their applicability as supervised targets for verbatim transcription. Rather than using subtitles as direct supervision signals, our method reimagines them as context-rich prompts. This design enables the model to handle discrepancies between spoken audio and subtitle text. Instead, generated pseudo transcripts become the primary targets, with subtitles acting as guiding cues for iterative refinement. To further enhance the process, we introduce a weighted attention mechanism that emphasizes relevant subtitle tokens during inference. Our experiments demonstrate significant improvements in transcription accuracy, highlighting the effectiveness of the proposed method in refining transcripts. These enhanced pseudo-labeled datasets provide high-quality foundational resources for training robust ASR systems.