# PiDA: Phonetically-Informed Data Augmentation for Robust Vietnamese Speech Translation

arXiv: https://arxiv.org/abs/2606.12911 · HTML: https://mlsanity.com/paper/2606.12911/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 40 references that are in the map.
- First posted on arXiv 11 Jun 2026.

## Builds on (most cited first, 22 of 22)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech](https://mlsanity.com/paper/2106.06103/) (2021)
- [FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech](https://mlsanity.com/paper/2205.12446/) (2022)
- [Synthetic and Natural Noise Both Break Neural Machine Translation](https://mlsanity.com/paper/1711.02173/) (2017)
- [ByT5 model for massively multilingual grapheme-to-phoneme conversion](https://mlsanity.com/paper/2204.03067/) (2022)
- [Towards Robust Neural Machine Translation](https://mlsanity.com/paper/1805.06130/) (2018)
- [XPhoneBERT: A Pre-trained Multilingual Model for Phoneme Representations for Text-to-Speech](https://mlsanity.com/paper/2305.19709/) (2023)
- [Robust Neural Machine Translation for Clean and Noisy Speech Transcripts](https://mlsanity.com/paper/1910.10238/) (2019)
- [PhoWhisper: Automatic Speech Recognition for Vietnamese](https://mlsanity.com/paper/2406.02555/) (2024)
- [PhoMT: A High-Quality and Large-Scale Benchmark Dataset for Vietnamese-English Machine Translation](https://mlsanity.com/paper/2110.12199/) (2021)
- [MEDSAGE: Enhancing Robustness of Medical Dialogue Summarization to ASR Errors with LLM-generated Synthetic Dialogues](https://mlsanity.com/paper/2408.14418/) (2024)
- [PWESuite: Phonetic Word Embeddings and Tasks They Facilitate](https://mlsanity.com/paper/2304.02541/) (2023)
- [MTet: Multi-domain Translation for English and Vietnamese](https://mlsanity.com/paper/2210.05610/) (2022)
- [Improving Vietnamese-English Medical Machine Translation](https://mlsanity.com/paper/2403.19161/) (2024)
- [ViMedCSS: A Vietnamese Medical Code-Switching Speech Dataset & Benchmark](https://mlsanity.com/paper/2602.12911/) (2026)
- [MultiMed-ST: Large-scale Many-to-many Multilingual Medical Speech Translation](https://mlsanity.com/paper/2504.03546/) (2025)

## Abstract (arXiv)

Cascaded speech translation (ST) systems suffer from error propagation when Automatic Speech Recognition (ASR) outputs incorrect transcripts. We present the first systematic categorization of ASR errors for Vietnamese ST, classifying substitution errors by phonetic cause and quantifying their impact on downstream Neural Machine Translation (NMT) performance using Linear Mixed-Effects Modelling. We confirm that most ASR substitution errors arise from phonetic confusions rather than random noise, and that these phonetic errors significantly degrade ST quality. Motivated by this finding, we propose Phonetically-Informed Data Augmentation (PiDA), which generates ASR-like corruptions by substituting words with phonetically similar alternatives using phonetic word embeddings. Fine-tuning on a PiDA-augmented version of FLEURS Vietnamese-English improves translation of erroneous ASR outputs (up to +2.04 BLEU over standard fine-tuning) while also slightly improving clean-text performance.