# Whisper-Based Speech Transcription from Videos Across Multiple Languages for Cross-Cultural Understanding

arXiv: https://arxiv.org/abs/2609.11772 · HTML: https://mlsanity.com/paper/2609.11772/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 27 references that are in the map.
- First posted on arXiv 10 Sep 2026.

## Builds on (most cited first, 12 of 12)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [WhisperX: Time-Accurate Speech Transcription of Long-Form Audio](https://mlsanity.com/paper/2303.00747/) (2023)
- [Powerset multi-class cross entropy loss for neural speaker diarization](https://mlsanity.com/paper/2310.13025/) (2023)
- [Survey of Cultural Awareness in Language Models: Text and Beyond](https://mlsanity.com/paper/2411.00860/) (2024)
- [NormSAGE: Multi-Lingual Multi-Cultural Norm Discovery from Conversations On-the-Fly](https://mlsanity.com/paper/2210.08604/) (2022)
- [LoRA-Whisper: Parameter-Efficient and Extensible Multilingual ASR](https://mlsanity.com/paper/2406.06619/) (2024)
- [NormDial: A Comparable Bilingual Synthetic Dialog Dataset for Modeling Social Norm Adherence and Violation](https://mlsanity.com/paper/2310.14563/) (2023)
- [Open ASR Leaderboard: Towards Reproducible and Transparent Multilingual and Long-Form Speech Recognition Evaluation](https://mlsanity.com/paper/2510.06961/) (2025)
- [JTubeSpeech: corpus of Japanese speech collected from YouTube for speech recognition and speaker verification](https://mlsanity.com/paper/2112.09323/) (2021)
- [Measuring Social Norms of Large Language Models](https://mlsanity.com/paper/2404.02491/) (2024)
- [Measuring the Effect of Transcription Noise on Downstream Language Understanding Tasks](https://mlsanity.com/paper/2502.13645/) (2025)
- [MINDS: A Cross-cultural Dialogue Corpus for Social Norm Classification and Adherence Detection](https://mlsanity.com/paper/2511.09918/) (2025)

## Abstract (arXiv)

Cross-cultural understanding has become increasingly important in today's highly connected, cross-national world. The success of LLM-based technologies is now driving the development of automated tools to aid understanding for nonnative people trying to succeed in cross-cultural environments. Building such automated tools is often done by leveraging in-thewild text, audio, and video data. This paper presents techniques for improving speech recognition-based transcript creation in multiple languages from videos to better train these automated tools. The focus is on processes and speech tools that can easily be used by cross-cultural tool builders without requiring deep speech processing expertise. Using publicly available videos from YouTube and Whisper-based tools, average transcription error rate across seven languages (Spanish, Japanese, Korean, Mandarin, Turkish, Russian, and Hebrew) of 30% are observed. With a modest amount of fine-tuning data, the average error rate can be reduced to 20% making such output much more usable for downstream processing. Speech and metadata associated with these videos that can be used by the community to further refine these experiments are released as well.