# Generating Human Readable Transcript for Automatic Speech Recognition with Pre-trained Language Model

arXiv: https://arxiv.org/abs/2102.11114 · HTML: https://mlsanity.com/paper/2102.11114/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 19 references that are in the map.
- First posted on arXiv 22 Feb 2021.

## Builds on (most cited first, 8 of 8)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions](https://mlsanity.com/paper/1712.05884/) (2017)
- [Unified Language Model Pre-training for Natural Language Understanding and Generation](https://mlsanity.com/paper/1905.03197/) (2019)
- [The Microsoft 2017 Conversational Speech Recognition System](https://mlsanity.com/paper/1708.06073/) (2017)
- [Correction of Automatic Speech Recognition with Transformer Sequence-to-sequence Model](https://mlsanity.com/paper/1910.10697/) (2019)
- [Improving Readability for Automatic Speech Recognition Transcription](https://mlsanity.com/paper/2004.04438/) (2020)
- [Sequence-to-sequence Pre-training with Data Augmentation for Sentence Rewriting](https://mlsanity.com/paper/1909.06002/) (2019)

## Built on by (most cited first, 2 of 2)

- [Pretrained Language Models for Text Generation: A Survey](https://mlsanity.com/paper/2201.05273/) (2022)
- [LT-LM: a novel non-autoregressive language model for single-shot lattice rescoring](https://mlsanity.com/paper/2104.02526/) (2021)

## Abstract (arXiv)

Modern Automatic Speech Recognition (ASR) systems can achieve high performance in terms of recognition accuracy. However, a perfectly accurate transcript still can be challenging to read due to disfluency, filter words, and other errata common in spoken communication. Many downstream tasks and human readers rely on the output of the ASR system; therefore, errors introduced by the speaker and ASR system alike will be propagated to the next task in the pipeline. In this work, we propose an ASR post-processing model that aims to transform the incorrect and noisy ASR output into a readable text for humans and downstream tasks. We leverage the Metadata Extraction (MDE) corpus to construct a task-specific dataset for our study. Since the dataset is small, we propose a novel data augmentation method and use a two-stage training strategy to fine-tune the RoBERTa pre-trained model. On the constructed test set, our model outperforms a production two-step pipeline-based post-processing method by a large margin of 13.26 on readability-aware WER (RA-WER) and 17.53 on BLEU metrics. Human evaluation also demonstrates that our method can generate more human-readable transcripts than the baseline method.