# Phoneme-BERT: Joint Language Modelling of Phoneme Sequence and ASR Transcript

arXiv: https://arxiv.org/abs/2102.00804 · HTML: https://mlsanity.com/paper/2102.00804/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 19 references that are in the map.
- First posted on arXiv 1 Feb 2021.

## Builds on (most cited first, 8 of 8)

- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Ups and Downs: Modeling the Visual Evolution of Fashion Trends with One-Class Collaborative Filtering](https://mlsanity.com/paper/1602.01585/) (2016)
- [Towards end-to-end spoken language understanding](https://mlsanity.com/paper/1802.08395/) (2018)
- [Joint Contextual Modeling for ASR Correction and Language Understanding](https://mlsanity.com/paper/2002.00750/) (2020)
- [Jointly Encoding Word Confusion Network and Dialogue Context with BERT for Spoken Language Understanding](https://mlsanity.com/paper/2005.11640/) (2020)
- [On the Choice of Modeling Unit for Sequence-to-Sequence Speech Recognition](https://mlsanity.com/paper/1902.01955/) (2019)
- [A systematic comparison of grapheme-based vs. phoneme-based label units for encoder-decoder-attention models](https://mlsanity.com/paper/2005.09336/) (2020)

## Built on by (most cited first, 21 of 21)

- [Error Correction in ASR using Sequence-to-Sequence Models](https://mlsanity.com/paper/2202.01157/) (2022)
- [From Babble to Words: Pre-Training Language Models on Continuous Streams of Phonemes](https://mlsanity.com/paper/2410.22906/) (2024)
- [PSSAT: A Perturbed Semantic Structure Awareness Transferring Method for Perturbation-Robust Slot Filling](https://mlsanity.com/paper/2208.11508/) (2022)
- [ed-cec: improving rare word recognition using asr postprocessing based on error detection and context-aware error correction](https://mlsanity.com/paper/2310.05129/) (2023)
- [ASR Error Correction with Constrained Decoding on Operation Prediction](https://mlsanity.com/paper/2208.04641/) (2022)
- [PANCETTA: Phoneme Aware Neural Completion to Elicit Tongue Twisters Automatically](https://mlsanity.com/paper/2209.06275/) (2022)
- [ML-LMCL: Mutual Learning and Large-Margin Contrastive Learning for Improving ASR Robustness in Spoken Language Understanding](https://mlsanity.com/paper/2311.11375/) (2023)
- [Mitigating the Linguistic Gap with Phonemic Representations for Robust Cross-lingual Transfer](https://mlsanity.com/paper/2402.14279/) (2024)
- [What BERT Based Language Models Learn in Spoken Transcripts: An Empirical Study](https://mlsanity.com/paper/2109.09105/) (2021)
- [Small Language Models Also Work With Small Vocabularies: Probing the Linguistic Abilities of Grapheme- and Phoneme-Based Baby Llamas](https://mlsanity.com/paper/2410.01487/) (2024)
- [Contrastive Learning for Improving ASR Robustness in Spoken Language Understanding](https://mlsanity.com/paper/2205.00693/) (2022)
- [Domain Adaptation in Intent Classification Systems: A Review](https://mlsanity.com/paper/2404.14415/) (2024)
- [PATCorrect: Non-autoregressive Phoneme-augmented Transformer for ASR Error Correction](https://mlsanity.com/paper/2302.05040/) (2023)
- [End-to-End Evaluation of a Spoken Dialogue System for Learning Basic Mathematics](https://mlsanity.com/paper/2211.03511/) (2022)
- [Elevating Code-mixed Text Handling through Auditory Information of Words](https://mlsanity.com/paper/2310.18155/) (2023)
- [wav2graph: A Framework for Supervised Learning Knowledge Graph from Speech](https://mlsanity.com/paper/2408.04174/) (2024)
- [PMF-CEC: Phoneme-augmented Multimodal Fusion for Context-aware ASR Error Correction with Error-specific Selective Decoding](https://mlsanity.com/paper/2506.11064/) (2025)
- [How Tokenization Limits Phonological Knowledge Representation in Language Models and How to Improve Them](https://mlsanity.com/paper/2604.17105/) (2026)
- [Building Robust Spoken Language Understanding by Cross Attention between Phoneme Sequence and ASR Hypothesis](https://mlsanity.com/paper/2203.12067/) (2022)
- [Learning Semantic Information from Raw Audio Signal Using Both Contextual and Phonetic Representations](https://mlsanity.com/paper/2402.01298/) (2024)
- [Handling Korean Out-of-Vocabulary Words with Phoneme Representation Learning](https://mlsanity.com/paper/2507.04018/) (2025)

## Abstract (arXiv)

Recent years have witnessed significant improvement in ASR systems to recognize spoken utterances. However, it is still a challenging task for noisy and out-of-domain data, where substitution and deletion errors are prevalent in the transcribed text. These errors significantly degrade the performance of downstream tasks. In this work, we propose a BERT-style language model, referred to as PhonemeBERT, that learns a joint language model with phoneme sequence and ASR transcript to learn phonetic-aware representations that are robust to ASR errors. We show that PhonemeBERT can be used on downstream tasks using phoneme sequences as additional features, and also in low-resource setup where we only have ASR-transcripts for the downstream tasks with no phoneme information available. We evaluate our approach extensively by generating noisy data for three benchmark datasets - Stanford Sentiment Treebank, TREC and ATIS for sentiment, question and intent classification tasks respectively. The results of the proposed approach beats the state-of-the-art baselines comprehensively on each dataset.