# POWSM: A Phonetic Open Whisper-Style Speech Foundation Model

arXiv: https://arxiv.org/abs/2510.24992 · HTML: https://mlsanity.com/paper/2510.24992/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 72 references that are in the map.
- First posted on arXiv 28 Oct 2025.

## Builds on (most cited first, 30 of 30)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale](https://mlsanity.com/paper/2111.09296/) (2021)
- [ESPnet: End-to-End Speech Processing Toolkit](https://mlsanity.com/paper/1804.00015/) (2018)
- [Scaling Speech Technology to 1,000+ Languages](https://mlsanity.com/paper/2305.13516/) (2023)
- [Language-Specific Neurons: The Key to Multilingual Capabilities in Large Language Models](https://mlsanity.com/paper/2402.16438/) (2024)
- [Universal Phone Recognition with a Multilingual Allophone System](https://mlsanity.com/paper/2002.11800/) (2020)
- [Simple and Effective Zero-shot Cross-lingual Phoneme Recognition](https://mlsanity.com/paper/2109.11680/) (2021)
- [OWSM v3.1: Better and Faster Open Whisper-Style Speech Models based on E-Branchformer](https://mlsanity.com/paper/2401.16658/) (2024)
- [Whisper-AT: Noise-Robust Automatic Speech Recognizers are Also Strong General Audio Event Taggers](https://mlsanity.com/paper/2307.03183/) (2023)
- [speechocean762: An Open-Source Non-native English Speech Corpus For Pronunciation Assessment](https://mlsanity.com/paper/2104.01378/) (2021)
- [ByT5 model for massively multilingual grapheme-to-phoneme conversion](https://mlsanity.com/paper/2204.03067/) (2022)
- [Can Whisper perform speech-based in-context learning?](https://mlsanity.com/paper/2309.07081/) (2023)
- [Analyzing Hidden Representations in End-to-End Automatic Speech Recognition Systems](https://mlsanity.com/paper/1709.04482/) (2017)
- [OWSM v4: Improving Open Whisper-Style Speech Models via Data Scaling and Cleaning](https://mlsanity.com/paper/2506.00338/) (2025)
- [Phonotactic Complexity and its Trade-offs](https://mlsanity.com/paper/2005.03774/) (2020)
- [ZIPA: A family of efficient models for multilingual phone recognition](https://mlsanity.com/paper/2505.23170/) (2025)
- [Whistle: Data-Efficient Multilingual and Crosslingual Speech Recognition via Weakly Phonetic Supervision](https://mlsanity.com/paper/2406.02166/) (2024)
- [OWLS: Scaling Laws for Multilingual Speech Recognition and Translation Models](https://mlsanity.com/paper/2502.10373/) (2025)
- [Universal Automatic Phonetic Transcription into the International Phonetic Alphabet](https://mlsanity.com/paper/2308.03917/) (2023)
- [Leveraging Allophony in Self-Supervised Speech Models for Atypical Pronunciation Assessment](https://mlsanity.com/paper/2502.07029/) (2025)
- [A Crowdsourced Open-Source Kazakh Speech Corpus and Initial Speech Recognition Baseline](https://mlsanity.com/paper/2009.10334/) (2020)
- [The taste of IPA: Towards open-vocabulary keyword spotting and forced alignment in any language](https://mlsanity.com/paper/2311.08323/) (2023)
- [Allophant: Cross-lingual Phoneme Recognition with Articulatory Attributes](https://mlsanity.com/paper/2306.04306/) (2023)
- [Subword Dictionary Learning and Segmentation Techniques for Automatic Speech Recognition in Tamil and Kannada](https://mlsanity.com/paper/2207.13331/) (2022)
- [Wav2Gloss: Generating Interlinear Glossed Text from Speech](https://mlsanity.com/paper/2403.13169/) (2024)
- [In-Context Learning Boosts Speech Recognition via Human-like Adaptation to Speakers and Language Varieties](https://mlsanity.com/paper/2505.14887/) (2025)
- [Tusom2021: A Phonetically Transcribed Speech Dataset from an Endangered Language for Universal Phone Recognition Experiments](https://mlsanity.com/paper/2104.00824/) (2021)
- [PolyIPA -- Multilingual Phoneme-to-Grapheme Conversion Model](https://mlsanity.com/paper/2412.09102/) (2024)
- [Speech-to-Text Translation with Phoneme-Augmented CoT: Enhancing Cross-Lingual Transfer in Low-Resource Scenarios](https://mlsanity.com/paper/2505.24691/) (2025)

## Built on by (most cited first, 14 of 14)

- [An Empirical Recipe for Universal Phone Recognition](https://mlsanity.com/paper/2603.29042/) (2026)
- [PRiSM: Benchmarking Phone Realization in Speech Models](https://mlsanity.com/paper/2601.14046/) (2026)
- [Multilingual Dysarthric Speech Assessment Using Universal Phone Recognition and Language-Specific Phonemic Contrast Modeling](https://mlsanity.com/paper/2601.21205/) (2026)
- [HuPER: A Human-Inspired Framework for Phonetic Perception](https://mlsanity.com/paper/2602.01634/) (2026)
- [OmniACBench: A Benchmark for Evaluating Context-Grounded Acoustic Control in Omni-Modal Models](https://mlsanity.com/paper/2603.23938/) (2026)
- [Scaling Human and G2P Supervision for Robust Phonetic Transcription](https://mlsanity.com/paper/2606.16019/) (2026)
- [ReNikud: Audio-Supervised Hebrew Grapheme-to-Phoneme Conversion](https://mlsanity.com/paper/2606.20179/) (2026)
- [BranchShine: Compact Raw-Audio-to-IPA Transcription with a RoPE E-Branchformer Encoder](https://mlsanity.com/paper/2606.22824/) (2026)
- [Quantifying the Generation Modality Gap in Speech-Text Language Models](https://mlsanity.com/paper/2609.14743/) (2026)
- [Linear Script Representations in Speech Foundation Models Enable Zero-Shot Transliteration](https://mlsanity.com/paper/2601.02906/) (2026)
- [Phone Segmentation and Recognition through Phonological Activation Mapping](https://mlsanity.com/paper/2607.09020/) (2026)
- [BranchShine-CR: Compact Multilingual IPA Transcription with Self-Conditioned CTC and Consistency Regularization](https://mlsanity.com/paper/2609.29069/) (2026)
- [Training-Free Pronunciation Transcription via Text-Constrained Acoustic Rescoring](https://mlsanity.com/paper/2609.30924/) (2026)
- [SHAMS: An Audio-Grounded Pronunciation Benchmark for Levantine Arabic](https://mlsanity.com/paper/2610.01427/) (2026)

## Abstract (arXiv)

Recent advances in spoken language processing have led to substantial progress in phonetic tasks such as automatic speech recognition (ASR), phone recognition (PR), grapheme-to-phoneme conversion (G2P), and phoneme-to-grapheme conversion (P2G). Despite their conceptual similarity, these tasks have largely been studied in isolation, each relying on task-specific architectures and datasets. In this paper, we introduce POWSM (Phonetic Open Whisper-style Speech Model), the first unified framework capable of jointly performing multiple phone-related tasks. POWSM enables seamless conversion between audio, text (graphemes), and phones, opening up new possibilities for universal and low-resource speech processing. Our model outperforms or matches specialized PR models of similar size (Wav2Vec2Phoneme and ZIPA) while jointly supporting G2P, P2G, and ASR. Our training data, code and models are released to foster open science.